<p>I spent the last 6 years of my career at Uber across both Finance and Business leadership roles. In this time, I witnessed first hand how AI and ML drove improved customer experiences, more efficient operations, and will be instrumental in unlocking a future with self-driving cars and trucks.&nbsp;</p><p>As I considered joining Scale AI, I got deeper into the latest trends and potential of AI. I have become even more excited about the many applications of AI and AI’s ability to improve decision making, drive productivity leaps, as well as deliver improved customer experiences and outcomes across every industry and domain on the planet.&nbsp;</p><p>What I found in Alex and his team at Scale is a vision to help all companies and the U.S. government realize the potential of AI. Scale’s mission is to accelerate the development of AI applications and to do it by first tackling one of the hardest problems in AI, better data. Scale started by supporting the autonomous vehicle industry but has since expanded across many industries and applications. They have built products and solutions that support customers at the leading edge of AI as well as companies that are just getting started. In short, this convinced me that the opportunity and potential are unparalleled.&nbsp;</p><p>As I spent more time getting to know Alex, we talked about what he was looking for in a CFO. We agreed that Scale needs a business-minded finance partner to foster continued, rapid growth of the Finance function that enables the company through increased strategic alignment, business model evolution, and increased operational discipline.&nbsp;</p><p>Today, I am excited to announce that I am joining Scale as their first CFO to help the team deliver on their vision of building and scaling what will be one of the most impactful enterprise companies in the world by accelerating the adoption of AI. I will be leading Finance, Accounting, Corporate Development, and Strategic Partnerships. We are hiring across the team. Come join us!&nbsp;</p><p><br></p><p><br></p>

Scale Welcomes Dennis Cinelli as First CFO

Why I Joined Scale

<p><em style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">At Scale, we have a handful of internal Employee Resource Groups (ERGs) that connect Scaliens around shared identities. That means we spend a lot of time thinking about lowering barriers, leveling the playing field, and fostering communities of belonging and shared interests. For members of the LGBTQ+ community, having space at work to be authentic and open is so important. And, it’s part of why we built Scale Out, where Scaliens rely on each other to create a community of inclusion.</em></p><p><em style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">Maverick Eason, Office Coordinator at SFHQ and Co-chair of Scale Out, shares their experience at Scale and the history of the Pink Triangle project.</em></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);"><img src="https://lh4.googleusercontent.com/vWM13AfeZMo1LX_inQDmjX9hKbrz4E8mkio7yyasFIyG-2kAeBkeVPp3WrZ8I5eVL8ljXuxEVwKLtWlrD_KOd3uTvNah45CuzSQ3vlEBxjhL-NKdyyXqtTmRPrbODpJCQytmjzdCJM80X261ig"></span></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">Every year since 1995, a pink triangle has been constructed at the top of Twin Peaks in San Francisco during the month of June for Pride. That first year, under the cover of darkness, community members climbed the Peaks and adorned it with the pink triangle– a symbol of gay power and pride for the LGBTQ+ community. Where did this symbol originate? Why a pink triangle?</span></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">During World War 2, Nazis would force people to wear different </span><span style="color: rgb(255, 255, 255);">signifiers</span><span style="color: rgb(255, 255, 255); background-color: rgb(255, 255, 255);"> of who they were– yellow stars for Jews, blue for immigrants, and pink triangles for homosexual men. The pink triangles were the largest and brightest of these patches, and those wearing them received the most severe brutality from both Nazis and other prisoners. When people were released from the concentration camps, many of those wearing the pink triangle were still kept imprisoned, some until as late as the 1970’s.&nbsp;</span></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">In the 1980’s, as the AIDS crisis was killing off record numbers of gay men and other LGBT people, the pink triangle was used on posters to call attention to the pandemic that the United States largely ignored– the symbol was adopted by ACT UP, a grassroots organization that works to end the pandemic and to help those living with HIV/AIDS. The pink triangle continues to be not only a symbol of power, but also serves as a reminder for what hatred and intolerance can do to us all.&nbsp;</span></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);">The pink triangle installation has developed into a San Francisco community event that is kicked off the first day of June, and includes a lighting ceremony by the mayor and other local politicians, organizers, and volunteers. I had the privilege of volunteering on May 26th as a site ambassador, along with the marketing and business developer of TransClinique, where we talked with visitors at the site and told them about the history and importance of the installation. Starting June 1st at sunset, the pink triangle will be visible every day at the top of Twin Peaks through the entire month. It serves as a reminder to us all of the impact of hatred, ignorance, and intolerance, and also the strength and power that the LGBTQ+ community continues to demonstrate every day across the globe.&nbsp;</span></p><p><span style="background-color: rgb(255, 255, 255); color: rgb(255, 255, 255);"><img src="https://lh5.googleusercontent.com/WdwUpmz36oS9ctweHy3MRU-sOvClflsQgZs4ThqvTcatMhkrVQUZgX8u9t-CrWyajt7i08sD5JFcczD0p20a3LFMVhOOFtBku6MOFrqU5lzbJpQ7I89WsSl1lkzByamEZcO10Z3w8sR3ri-LeA"></span></p>

Maverick Eason, Office Coordinator at SFHQ and Co-chair of Scale Out, shares their experience at Scale and the history of the Pink Triangle project.

The Pink Triangle  

Honoring Pride: Scale Out & The Pink Triangle  

<p><em style="background-color: transparent; color: rgb(255, 255, 255);">This Memorial Day, join us on Jacob's journey as he shares his experiences with the costs of war.</em></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">My college application process was based on a single announcement. During a senior-year English class, my high school principal walked in and announced two passenger airplanes had struck the World Trade Center buildings. While West Point was already my top choice, I went home and threw away applications I had started to other colleges. Witnessing 9/11 was a formative experience for me and my response to it was fully naive. At the time, I didn't know much about al-Qaeda. I likely couldn’t point out Afghanistan on a map. And I certainly couldn’t have a discussion about different sects within Islam. My framework for responding was just as naively simple: do something.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">Four days after graduating high school, I arrived at West Point. Our freshman class underwent our first summer of military training with a palpable sense of urgency that we </span><em style="background-color: transparent; color: rgb(255, 255, 255);">would</em><span style="background-color: transparent; color: rgb(255, 255, 255);"> – not </span><em style="background-color: transparent; color: rgb(255, 255, 255);">might</em><span style="background-color: transparent; color: rgb(255, 255, 255);"> – go to war. As the academic year started, I caught my first glimpse of the war’s toll in an unlikely place – the cadet mess hall. West Point’s mess hall is a cavernous and historic cafeteria of sorts, a place seemingly pulled straight from Hogwarts, where the operational feat of feeding 4,000 hungry cadets in 20 minutes happens not once, but three times a day. Seconds before lunch, the cacophony of noises that filled the mess hall was exchanged with perfect silence as a cadet leader barked daily instructions from a central, two-story platform. These instructions were usually about something mundane like uniform or schedule changes. Yet some days the instructions were replaced with a solemn announcement that a West Point graduate died during combat operations in Afghanistan.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">These announcements increased toward the end of my freshman year as the United States launched another campaign in the Global War on Terror with the invasion of Iraq. By the time I was a senior, some announcements were graduates I knew well. First Lieutenant Gary Avery was my teammate from an arduous cadet military skills competition. Major Bill Hecker was my eccentric freshman year English composition instructor who left behind his wife to raise their four children. As a 19-year-old kid, I struggled to make sense of news like this. I would hang my head for the rest of the meal, then return to my routine of cramming for an afternoon exam.</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">While I no longer heard lunchtime announcements after I graduated from West Point, the news of more deaths did not stop. Through text messages from friends, I would hear about other graduates I knew – like Emily Perez, Tom Martin, and Matt Ferrara – who were killed in combat. Eventually, it was my classmates’ turn to deploy to Iraq and Afghanistan. Tim Cunningham and Nick Dewhirst were the first two from my class not to make it back home. During my second combat deployment as a Special Forces team leader in southeastern Afghanistan, I heard frantic radio traffic detailing an insider attack in an adjacent province that killed three American servicemembers. Later that day I would learn one of those killed was my West Point classmate and fellow Special Forces team leader Andrew-Pedersen Keel, better known as “PK.”&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">Even after I returned home from Afghanistan, the announcements continued. The flow of Taliban from Pakistan increased into the area my team had previously secured only months prior. These new fighters killed 11 Afghan police at one of their checkpoints in the middle of the night. A few weeks later, the Taliban fired a rocket at an American convoy in the same area, killing a soldier previously assigned to my team and my best interpreter. The soldier left behind three children and the interpreter was only a few weeks away from going home on his last of several deployments to Afghanistan.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">A couple years later while I was in graduate school, I woke up to another announcement, this time in the New York Times. A Taliban suicide bomber had walked into a crowded volleyball match in the exact area where my team had been in Afghanistan, killing over 40 people, eight of them local police my team had trained. I remember watching a volleyball tournament there two years prior – the first since the Taliban had outlawed all sporting events. It was one achievement I used to rationalize the costs of the war. I thought this progress would never be reversed, but it was erased instantly.</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">I still struggle to make sense of the many announcements that defined my transition into adulthood and my early professional life. I think about the families of those killed for whom there is no such thing as a return to normal. I often feel guilty for making it back from war alive or for tactical decisions I made that contributed to the loss of life. I don’t understand how I escaped unscathed from the ambushes and other violence I encountered. The role of chance in war is maddening and absurd.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">Aggregating the costs of war is also an exercise in absurdity. The Watson Institute at Brown University reports that over 7,000 American service members have been killed in post-9/11 combat operations. Over 30,000 American service members – four times the number of those who died in combat – have taken their own lives over the same time period. Beyond the U.S., Western allies who sent troops to support American wars account for roughly 1,000 more deaths. I’m acutely aware of the estimated 200,000 deaths of partner military and national police forces in places like Iraq, Afghanistan, and Syria. Fighting alongside them, I quickly realized how remarkable their bravery and sacrifice were, especially considering how under-resourced they were by their governments. Even more hidden from public view are the estimated 300,000 civilians who died violent deaths directly related to the wars. This doesn’t include the indirect impact on civilians, like those who lost access to basic resources or were displaced from their homes as refugees. Financially, the Watson Institute estimates the U.S. spent nearly $8 trillion related to post-9/11 wars.</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">You might have strong reactions as you process the costs of these wars. You might bristle with pride at those who gave the last full measure of devotion for their country. You might feel angry at the cost of America’s wars and view them as misadventures marked by hubris and strategic failures. You might feel a sense of guilt as you try to understand an experience from which you are so detached. All of these reactions aren’t merely okay – they are fundamentally human. Whatever your reaction to these costs of war, I encourage you to take two actions as Memorial Day approaches.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">The first action is reflection. I find it useful to reflect on an obvious, yet unsettling fact: we are all finite. Even though much of my adult life was filled with announcements reminding me of life’s brevity and fragility, it is not natural to linger on this reality for more than a few moments. Set aside time to reflect on this and ask yourself: “What causes are worth my life?”, “What acts of service will I undertake with the finite time I have left?”, and “In what ways am I living sacrificially for others?” Reflecting on questions like these can be helpful in living an examined life to deliberately grow into the person you hope to become.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">The second action is remembrance. My most powerful experience of remembrance came while I was serving as an instructor at West Point and visited its historic cemetery on Memorial Day a few years ago. When I approached a row of tombstones of West Point alumni who graduated around the time I was a cadet, I paused on one that belonged to First Lieutenant Laura Walker. I noticed the date of her death and realized she was one of the announcements I heard at lunch during my junior year as a cadet. As I gazed at some photos placed near her tomb, two people walked toward me who had been sitting under a nearby tree. They were Laura’s parents. They make an annual pilgrimage to West Point’s cemetery on Memorial Day to meet others who come across Laura in the cemetery and to share her stories with them. What struck me the most while talking with them were the human terms – like “goofball” – they used to describe Laura. There was no mention of the word “hero,” just relatable stories that highlighted Laura’s similarities to the rest of us.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">In that spirit, my act of remembrance for this Memorial Day was to learn more about the stories of those who were announcements in my life. I learned that Gary Avery liked sewing and even started a nonprofit for Iraqi orphans while he was a cadet. I learned that Bill Hecker was a Saint Louis Cardinals fanatic and loved gathering his four kids on the couch before bedtime to give dramatic readings from Little House on the Prairie. I learned that Emily Perez was known during her days as a cadet for jingling her tambourine through the hallways of the barracks on Sunday mornings as she walked to a Baptist church service.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">I encourage you to do the same kind of remembrance. As we remember the fallen for the ordinary people they were – for how similar they are to you and me – we realize that the ordinary human agency they had to make selfless choices with their lives is also available to the rest of us.&nbsp;</span></p><h3 style="text-align: center;"><span style="background-color: transparent; color: rgb(255, 255, 255);">About Veterans@Scale</span></h3><p><span style="background-color: transparent; color: rgb(255, 255, 255);">We’re investing in a diverse and inclusive workforce at Scale, and our workforce is as diverse as the customers we serve. We have different backgrounds and varied experiences. But wherever we work, whatever we do, we share a single set of values that unite our company.&nbsp;</span></p><p><span style="background-color: transparent; color: rgb(255, 255, 255);">One of the pillars of our diversity, inclusion, and belonging efforts is Employee Resource Groups (ERGs). We currently have five ERGs, and each one plays a vital role in ensuring that those different communities have a voice and can actively influence our company policies and culture. Today, we’re happy to shine a spotlight on Veterans@Scale and hear from one of its veteran members, Jacob Sheehan.</span></p><h3 style="text-align: center;"><span style="background-color: transparent; color: rgb(255, 255, 255);">About Jacob</span></h3><p><span style="background-color: transparent; color: rgb(255, 255, 255);">Jacob leads the Federal labeling workforce at Scale as the Head of Federal Supply. He served in the U.S. Army as a Special Forces Officer and an assistant professor of Economics at the United State Military Academy (West Point). He is a graduate of Stanford’s Graduate School of Business and a Fulbright Scholar to the United Kingdom.</span></p>

This Memorial Day, join us on Jacob's journey as he shares his experiences with the costs of war.

Memorial Day Reflection

Veterans@Scale: Reflection and Remembrance on Memorial Day

<p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: Tell us a little bit about yourself and your role at Scale.</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A: My name is Sasha, and I'm a Machine Learning Engineer here at Scale. I've been at Scale for about a year and eight months. Previously, I was doing a master's program in computer science at Stanford, where my concentration was in artificial intelligence.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Within my career at Scale, I've worn a lot of different hats. I first started working on the Nucleus team, in a backend software engineering role when the team was very young. It was&nbsp;Scale’s newest product at that point, and I helped build out a lot of Python SDK interfaces, and the backend schema for a lot of new features, including Autotag, which is the edge case detection feature.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">I later transitioned to building out other new products, including Scale Validate, which is the model scenario testing and validation product area.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Most recently, I've recently transitioned to our Machine Learning team full time, where I'm on the Content Understanding team. I'm basically helping to support automation in labeling projects associated with content categorization and curation type labeling projects.</span></p><p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: Could you give us a little bit of detail about your educational background, and how that led you to join Scale?</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A: My undergrad degree was in applied mathematics. Entering college, I didn't think of myself as a coding person at all. But during the years I was at Stanford, computer science was something that a lot of the undergraduate body talked about, and the entry level computer science classes were really fun. </span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">During my junior and senior years in college, I started taking some entry level machine learning classes. The first thing I ever learned about was logistic regression, and how you assign importance or attribution to different input features&nbsp;to answer social science questions. Like, is it age, or education, that contributes more to this downstream factor?</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">I thought that was really interesting, and really wanted to double down on that. So, that's what led me to stay for a fifth year master's, and specialize in AI. I worked as a research assistant briefly, in a sociology lab, where the objective of the sociology lab was to develop a dataset, documenting urban change. My job as a research assistant was to label data on AWS, Mechanical Turk. We had a lot of difficulty controlling quality, and getting annotations that were detailed enough to model the downstream tasks of trash detection and vandalism detection.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">So, I knew that data collection was a hair on fire problem for a lot of application-driven teams. When I interviewed at Scale, I felt like the mission, and the problems that the company was trying to solve, were very urgent and impactful.</span></p><p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: What attracted you to working at a hyper growth startup?</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A: When I was in college, I think a lot of my peers were talking about going to work at a big tech company; that was the prestigious choice. It was safe; it was a known outcome. At first, I was really focused on that, and I did a lot of internships at big companies.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">And then I got this career advice my sophomore junior year of college, from a woman at Grace Hopper, and her advice was get on a rocket ship. Basically, the idea was, if you could just put yourself in a place where there were interesting problems to solve, some of those were going to come your way.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">I really took that to heart, and my impression of myself started to shift toward somebody who really wanted to tackle big challenges and wear a lot of hats in their job. So, I joined Scale under the hypothesis that I was going to have an opportunity to wear a lot more hats and serve a lot of different functions at a very lean company.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">As a result of having the opportunity to take on difficult challenges, I've grown my skills at a very quick rate. And I think I gained seniority and experience, at a rate that far exceeds any previous work opportunity I had in the past.&nbsp;</span></p><p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: What's your advice to students, or other people, who are interested in careers in machine learning?</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A: First of all, I think it's an excellent career to be in, in that there are a lot of opportunities in the space right now. And the number of positions for machine learning engineers and machine learning-adjacent engineers, is growing really rapidly. I think there are also a lot of exciting developments, both on the research and applications side.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">One thing that fascinates me about machine learning is how closely it can relate to real-world problems. If you’re interested in sociology, there's probably a place for that type of modeling for you. Or, if you're interested in online advertising, there are a lot of different application areas to concentrate on.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">The other thing that I would recommend is really reading a lot. There are a lot of great resources online for practicing building deep learning models, whether it be through Kaggle challenges, or what have you. Also, a lot of research papers and academic work are publicly available.</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">One thing that I felt intimidated about when I started in the field was, how much there was to know, and how little of it I did know, as somebody that was early in their career. And I think making a habit of reading papers, and reading about approaches or problems that I didn't work on directly on a day-to-day basis, really increased my confidence.</span></p><p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: Has working at Scale shifted your perspective on AI or ML applications at all?</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A: Definitely. When I first started at Scale, what I was really interested in was machine learning modeling work, and the idea of experimentation–building different neural network architectures, and being very scientific about, whether A works better than B.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">One thing that's become really apparent, both working at Scale, and seeing the types of problems that our engineering teams face, but also working with Scale customers, is the importance of systems design within large scale machine learning systems. I’ve seen how difficult the problems of scalability, observability and overall reliability of machine learning systems are.&nbsp;</span></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">I really think that that's where a lot of innovation in the next 10 years is going to come from; not necessarily the modeling techniques themselves, but more, how you might integrate those modeling techniques into a larger scale operational framework.</span></p><p><strong style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">Q: What do you like to do outside of work?</strong></p><p><span style="color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);">A:&nbsp;I've recently gotten into cycling. And so, I spend a lot of my time outside, whether it's running or riding my bike in the Bay Area. I also love wheel pottery. So on the weekends, you'll find me at the pottery studio.</span></p>

Sasha Harrison details her experience as a Machine Learning Engineer.

Life at Scale

<p>Document AI is powered by machine learning (ML) models that process documents at high accuracy and speed. When these models are fine-tuned for a customer, they become even more accurate and regularly achieve near-perfect accuracy levels and do not require any human intervention. Scale engineers continuously work to improve these models by creating new tools to make sure Document AI is highly reliable for the most critical industries we serve. This blog will highlight one of those tools, a technique called “linting”, and how it helps make Document AI’s models more accurate.</p><h2>What is Linting?</h2><p>At Scale, linting is the practice of creating rules or developing machine learning models that automatically detect errors in training data used to create models. In the case of Document AI, we leverage two types of linters: simple, quick linters and more complex ML-based linters.&nbsp;</p><p>Here is an example of a simple linter comparing field types and their responses within a document.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8b843651dad4ba48f5831c07b48c6f2" width="755"></p><p>In the example above, the letter ‘O’ should not be in a numerical, monetary amount field and our field-type linter would flag this example for correction.</p><p>Another example of the simple quick linters is a linter that enforces known business rules. For example, a linter can ensure an invoice subtotal is equal to the sum of all line item totals.</p><h3><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F9e0a34e10d464195911c07b617ee27c7" width="574"></h3><p>In the example above, the subtotal on the right ($16) is inaccurate and our linter would flag this example for correction.</p><p>These simple linters are helpful where straightforward rules can easily be applied. Simple linters, while useful, can be implemented by most. What’s more unique about Scale Document AI is how we go beyond the mainstream methods and tools in the document industry to achieve high-quality.&nbsp;</p><p>Scenarios that require a higher level of capability or understanding are better suited to machine learning (ML)-powered linters instead of rule-based linters, which is Scale’s specialty. Our specialized ML linters are trained to perform specific tasks, such as learning to compare differences in a transcribed text and the visual representation of the text as it appears on the document’s page. In this example, when the linter finds a meaningful discrepancy, it automatically corrects it or flags the document for review by highly experienced and skilled reviewers. This human-in-the-loop review is optionally available based on customers’ needs and preferences.</p><p><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F626b6d1309fa4dc28c28d6d75279abb6"></p><p>In the example shown above, we trained an ML-based linter to understand the context to know that “Account #” on the bottom left refers to a bank account number as it is in a section with ACH instructions while “Account Number” on the top right is referring to a business account number. If “Account Number” in the example on the top right had been identified as a bank account number, the linter would flag this for correction.</p><p>ML linters are inherently more difficult to implement than rule-based linters as they have the same challenges as developing any other machine learning models. Developing an ML model is a statistical process prone to variation. That variation can manifest as unreliability or as overfitting. For example, unreliability can lead to an ML model producing wildly different predictions on two slightly different documents. With overfitting, the model learns the detail and noise in the training data and fails to perform on new data. In the case of linting, the ML models are trained on data with errors to help identify and flag other errors. The data used for training linters is inherently not of the highest quality; therefore it’s common that while the ML linter can still make decent suggestions, the model itself is not the most performant. This requires our ML team to place more restrictions and guardrails, like ensuring the confidence threshold is set high to only flag errors and make suggestions on things the model is fairly certain about. This prevents the model from producing too many unreliable linter suggestions that lead to human reviewers simply ignoring or not trusting these suggestions.&nbsp;</p><p>Developing and deploying ML linters is also an iterative process with a feedback loop. As we deploy ML linters for our document processing models, the document processing models improve upon being retrained with higher quality data. As the document processing models improve, it gives better first results as well as learning other errors to surface to the human reviewers.</p><h2>How Do Linters Improve Models?</h2><p>The power of machine learning to recognize patterns between input and output data is its greatest strength as well as its Achilles’ heel. If the training data contains errors, the model will learn the wrong task. Having the right linters in place increases the accuracy of machine learning models by ensuring the data used for training is of the highest quality.&nbsp;</p><p>For Document AI, we began by designing simple linters to check field types such as monetary amounts, quantities, and ZIP codes. We then expanded beyond these simple linters and implement ML-based linters to check for complex fields such as addresses with varying formats that we would need to parse and enforce.&nbsp;</p><p>To make things more concrete, let’s think about the use case of invoices. At first glance, it seems straightforward to recognize an invoice: they all list the products sold or the services rendered, the price of each, and how payment should be made. However, invoice layouts, details, and content vary widely from business to business. Some invoices are particularly formal, containing everything from company bank account numbers and line item descriptions to account manager names and contact information. On the other end of the spectrum, an informal email between longtime business partners can also serve as an invoice. The unstructured and ever-changing nature of invoices make it challenging to model comprehensively and accurately.</p><p>When building Document AI’s machine learning model for processing invoices, using linters resulted in a <strong>16.3% relative decrease in transcription errors in the training data</strong>. This relative decrease in transcription errors is crucial, as the misspelled text makes it impossible for our document parsing models to extract the right information. This linter also helped remove the need for an additional layer of human quality assurance (QA) review while still ensuring that training data is pristine.&nbsp;</p><p>Training our models on the higher quality training data led to a <strong>14.3% relative increase in our document processing model’s accuracy</strong>. This improvement translates to increased model generalizability and robustness, and more accurate models mean fewer manual reviews, providing additional value for our customers.&nbsp;</p><p>Linters make Document AI more accurate and hence deliver higher quality models to our customers who rely on high accuracy document processing.</p><h2>Parting Thoughts</h2><p>Scale Document AI distills insights from your messy, changing document data and delivers fast, accurate models tailored to your business without the need for templates. If you’re interested in learning more, take a look at our <a href="https://www.youtube.com/watch?v=jWCwVcQCA10" rel="noopener noreferrer" target="_blank">product video</a> or visit our <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">product page</a>.</p><p><em>Acknowledgments:</em></p><p><em>We thank Malcolm Greaves from Scale’s Document AI Machine Learning team for his experiments that laid the groundwork for this post and for frequent consultations throughout the writing of this blog.</em></p>

Scale Document AI

Document AI's linters improve model quality by ensuring common and hard-to-spot errors never make it into the training data.

Scale Document AI leverages ML Linting to Improve Document Processing

How ML Linting Meaningfully Improves Adaptive Document Processing at Scale

<p><em>Willow leads the Customer Operations team at Scale. She is a former director of logistics at Amazon.com, a graduate of MIT’s Leaders for Global Operations program, and a former captain in the U.S. Army’s Military Police Corps.</em></p>
<p>I am a transgender pioneer within the military and corporate world, with a career spanning 12 years in the U.S. Army, MIT, Amazon, and Scale. Here are some of the things I learned along the way.</p>
<p><strong>Why I decided to change my gender</strong></p>
<p>I had decided to change my gender while in a concrete shelter from rocket attacks right before returning from deployment in Afghanistan. Rockets in this context were not particularly threatening, but sitting in a concrete shelter at night waiting for the sirens to go off can naturally lead to introspection.</p>
<p>I had no intention of being a pioneer. It was a simple search for my own personal happiness. I felt like any other choice would be the wrong one. It was too much of a big and scary decision to think at all about how my career would shape out.</p>
<p>There were deliberate risks I took and risks that I didn’t expect and needed help with.</p>
<p>Back in the states, I was the first officer in my military unit to be out as bisexual after the repeal of “Don’t Ask, Don’t Tell” (DADT). This was a deliberate risk: my personal and professional lives didn’t intersect, but I thought that someone should be out as an example if other soldiers wanted to come out.</p>
<p>I still felt like I had a vulnerable secret, as being transgender was still prohibited, and my plans to transition after departure were carefully guarded even as I changed my legal name and started living as female while on leave. My commanding officer was overtly hostile towards the repeal of DADT. After one staff call in which he used “faggoty” as a term of derision, I grew concerned about whether or not I would be able to leave the Army on my own terms. I need not have been concerned, however. Senior members of the staff met with the commander over his comments, and he decided to leave the Army over the pace of change. Like me, but for different reasons, he was voting with his feet.</p>
<p>Returning to graduate school, I was the first openly transgender person in my program. I was one of two transgender students to tell my story to the school’s leaders in an ultimately successful bid to change insurance coverage policies for all students. Not all was respect and smooth sailing; in one emotionally crushing case, two teachers complained to my program admin about me using the restrooms that matched my gender identity (it seems strange to type this - much has changed in eight years). My friends calmed me down, and school administrators used the complaint as a teaching moment for the faculty.</p>
<p><strong>The only [other] in the room?</strong></p>
<p>As my civilian career advanced at Amazon and at Scale, I became more aware of how being transgender would shape my approach to work. No matter the realities of my professional role, I always feel like a bit of an outsider. It has been argued by folks more articulate and demonstrated in studies that diverse perspectives lead to better decisions, and clearer identification of risks. This is a good thing.</p>
<p>The downside to feeling like an outsider is always feeling the need to push myself harder, to prove myself better, and to hold myself to an extremely high internal standard - and to always fear slipping. This approach to work has helped me advance quickly, and made me bold, but it is exhausting at times and ultimately rooted in fear.</p>
<p>One of the factors in my recent decision to join Scale was that many company leaders identify with the LGBTQ community. I want to help build the future without worrying so much about how I fit in it. Taking a role at a smaller, fast-moving startup with less formal structure has been awesome. There’s a sense of purpose and mission and a rapid flow of ideas that I feel I can really be part of.</p>
<p>Transgender people have always existed in one way or another, before society’s own peculiarities and social constructs started calling it a “new thing.” I wrote earlier that I was the first openly transgender applicant at my grad school program. It is highly likely that there were transgender people there before, and it is likely that some of my classmates were transgender and simply hadn’t decided to transition yet. It is important to remember that an individual story cannot represent a community, especially with big variation in opportunity and privilege shaping peoples’ lives.</p>
<p><strong>Advice for folks feeling like outsiders</strong></p>
<p>There are a few things that I wish I’d known earlier in my career. One piece of advice I got from a friend has stuck with me: “You will always think about yourself a lot more than others think about you.” That feeling of awkwardness, that one thing you did - most people simply won’t notice or remember. Our memories are inherently selective. Learn from your mistakes but choose to remember the good parts.</p>
<p>The other thing I wish I’d understood better earlier is to not be afraid to pick a mission with purpose. Early on, I picked assignments that seemed like the best value for the company’s ability to save money or expand programs and to advance my career. I left jobs and teams that I enjoyed to seek out the external trappings of success. Many people do this, but the feeling of validation seemed so necessary when I was uncertain of my place in society. Finding a team and a voice that makes me feel at home and a mission that serves a purpose I believe in has ultimately been more gratifying. Now, what the people around me think matters more than what the world thinks. </p>
<p>Scale believes there is tremendous potential for AI technology to drive a revolution in true efficiency and productivity, across industries - this is a change for which I want a front row seat. I am glad I am able to take that seat feeling like I am no longer an outsider.</p> 

Scale Operations leader Willow Primack shares her personal experience for International Transgender Day of Visibility

Willow Primack shares her experience for International Transgender Day of Visibility

Trans Day of Visibility: My Experience at Scale

<h1 id="projectideasandinspiration">Project Ideas and Inspiration</h1>
<p>Scale recently sponsored Treehacks, Stanford University’s annual hackathon. During this fully virtual hackathon, nearly 1300 active hackers and 114 teams submitted projects for judging. Out of all teams, 21 submitted projects using Scale’s API.</p>
<p>The winning project for Best ML Hack was <a href="https://devpost.com/software/fine-tuned-heads">Fine Tuned Heads</a>, which proposed a new workflow to assist with the implementation of Computer Vision. Rather than training a single model from scratch, the encoding and fine-tuning aspects of a task were separated. The Fine Tuned Heads team created an automatic data collection pipeline using Flask to automatically execute scripts to grab .m3u8 URLs from Youtube, then used Scale Rapid to automate uploading and generating batches of ground truth labels for these images. According to the Fine Tuned Heads team, “this approach can be used to improve the efficiency of inference in ML tasks, and by engineering our self-supervised model with an efficient architecture and deploying on efficient hardware, we can reduce the carbon footprint by over 100x.''</p>
<p>An additional winner of the API Challenge was <a href="https://devpost.com/software/accecity">AcceCity</a>, which provides an ML-powered mapping platform that enables cities, urban planners, neighborhood associations, disability activists, and more to identify key areas to prioritize investment in. For their project, the AcceCity team first used the Google Maps API to send images of street views to their backend. They looked for handicapped parking, sidewalks, disability ramps, and crosswalks and used computer vision, by custom-fitting a zero shot learning model called CLIP from OpenAI, to automatically detect those objects from the images. They then tested the model using labeled data from the Scale Rapid API.</p>
<p>Other interesting projects include using Scale Rapid to label map data to promote sustainable living, and classify patient medical image data. </p>
<p>Beyond the examples from the Hackathon, one can use Scale Rapid for a variety of ML projects, including to <a href="https://scale.com/blog/rapid-cyclegan">anime-fy images</a>  and to train a <a href="https://scale.com/blog/imdb-rapid">sentiment analysis model for movie reviews</a>. Use our <a href="https://scale.com/open-datasets">open datasets</a> to draw further inspiration for your projects.</p>
<h1 id="howcanyouusescaletobuildasuccessfulmlhackathonproject">How can you use Scale to build a Successful ML Hackathon Project?</h1>
<p>One of the most challenging problems preventing teams from creating novel ML projects in a short timespan is the “Cold Start” problem, which is when an ML model cannot draw inferences for users or items about which it has not yet gathered sufficient information. Especially during the compressed timespan of a hackathon, obtaining high-quality training data for models is nearly impossible.</p>
<p>Scale Rapid is the fastest way to obtain production-level quality labels, with no data minimums. Using Scale Rapid, your hackathon team can experiment quickly by: </p>
<ol>
<li>Setting up projects in minutes and receiving initial data within hours</li>
<li>Iterating over edge cases and instructions by getting real-time feedback</li>
<li>Scale to production-level pipelines with precision quality control </li>
</ol>
<p>To get started, sign up for an account with your university email on our <a href="https://scale.com/rapid/university">University Program Page</a> to receive $250 in Rapid Credits, or check out our <a href="https://colab.research.google.com/drive/1XZjbVtA2oGOdMp0-v0E80ujMB8tbeh7x?usp=sharing">quick start python notebook</a>.</p>
<p>Rapid currently supports the following annotation types:</p>
<ul>
<li>General Image Annotation</li>
<li>2D Semantic Segmentation</li>
<li>Text Collection/Categorization</li>
<li>Document Transcription</li>
<li>Named Entity Recognition</li>
<li>Video Playback Annotation</li>
<li>Lidar Annotation</li>
</ul> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2ac2bcbb-1e7d-4268-2c77-bf6d2623c400/public" alt="Annotation Types Supported in Scale Rapid" />
        </figure><div class="markdown-module-blog"><p>If you have your own unlabeled data, add it to your project using local file or csv upload, import it from previous projects, or upload it from the cloud (S3, GCP, and Azure). Otherwise, check out our <a href="https://scale.com/open-datasets">open datasets</a>.</p>
<p>Once you receive task responses, visualize your data in Scale’s dataset management platform, Scale Nucleus, by navigating to the data tab under your project and clicking “Visualize in Nucleus”. Nucleus helps you understand the strengths and weaknesses in your dataset, analyze the long tail of your dataset, collaboratively edit labels, and explore metrics, failure cases, and confusion matrices linked to your training data. </p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/899fda4e-d4ab-4ed4-7ef4-5fa12e396200/public" alt="Scale Nucleus <> Scale Rapid Integration" />
        </figure><div class="markdown-module-blog"><p>Quickly find similar images using the “Autotag” feature in Nucleus, and further refine your set of images by  providing a few examples of positive and negative examples. After identifying this subset of your dataset, you can send it back for labeling through Scale Rapid.</p>
<p>Scale AI’s products help enable and strengthen a team’s ML development cycle immensely by delivering high quality data for any project. As you or your team continues to maintain and improve your project post-hackathon, use Scale AI for consistent, reliable data in order to create, validate, and maintain production for your high-performing ML models. </p></div>

Scale Rapid

Scale AI recently sponsored Treehacks, Stanford University’s annual hackathon. Explore how to use Scale Rapid for a successful ML hackathon.

Using Scale Rapid for Hackathon Projects

How to Use Scale for Your ML Hackathon Project

<h1>Background</h1>
<p>
  At Scale AI, we’re always looking for ways to apply machine learning (ML) in
  our internal systems to deliver the best results for our customers. We’ve
  previously written about Scale’s
  <a
    href="https://scale.com/blog/how-we-scale-machine-learning"
    rel="noopener noreferrer"
    target="_blank"
    >ML-powered pre-labeling</a
  >
  as well as our
  <a
    href="https://scale.com/blog/autosegment"
    rel="noopener noreferrer"
    target="_blank"
    >active tooling, like Autosegment</a
  >, which helps our Taskers label faster and at higher quality.
</p>
<p>
  In this piece, we discuss how we think about and incorporate some basic tenets
  of design thinking to develop ML systems that make a difference for our
  labeling pipeline.
  <a
    href="http://web.mit.edu/jrankin/www/engin_as_lib_art/Design_thinking.pdf"
    rel="noopener noreferrer"
    target="_blank"
    >Design thinking</a
  >
  is a mindset and set of practices which foster the collaboration required to
  solve problems in human-centered ways and was notably popularized by
  organizations such as IDEO. It’s useful for machine learning engineers who
  need to understand the full scope of how the system they are building will be
  used and are responsible for both business and operational outcomes. We’ll
  describe how design thinking techniques can be applied to scoping machine
  learning problems and introduce some concepts to help business stakeholders
  communicate more effectively with ML engineers.
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a41ed7a-4287-491b-6202-1f80a4c29a00/public"
    alt="Phases of Developing an ML System"
  />Idealized phases of developing an ML system. In practice, it’s not this
  linear.
</p>
<p>In our experience, the different phases of developing an ML system are:</p>
<ol>
  <li>Discovery - Locate a business or operational pain point</li>
  <li>
    Design - Map the different ways a new or existing model could fit into the
    extant pipeline
  </li>
  <li>
    Decision &amp; Alignment - Align with operational and engineering
    stakeholders on scope and resourcing
  </li>
  <li>
    Development, Integration, Deployment - Deliver the system. Build data
    pipelines, train models, integrate into the pipeline, etc.
  </li>
  <li>
    Measurement &amp; Experimentation - Identify the effect of the ML system as
    an intervention on the existing pipeline.
  </li>
  <li>
    Continuous improvement - Monitor and retrain to account for distribution
    shifts and new edge cases
  </li>
</ol>
<p>
  This is a descriptive mental model, not a prescriptive blueprint. In reality,
  a project is always moving between different phases and can often be in
  multiple phases at the same time. For example, measurement and development
  form a tight iteration loop. But, it’s still useful to think about these
  phases in isolation.
</p>
<p>
  Concepts from design thinking are mostly employed in the
  <strong>Discovery</strong> and <strong>Design</strong> phases when the problem
  statements are the most inchoate and ill-formed. In these phases, the need or
  specific outcome is communicated to an ML engineer and an initial strategy to
  address that need or deliver that outcome is designed. We focus on the
  Discovery and Design phases in this blog and cover the remaining phases
  briefly.
</p>
<h1>Phase 1: Discovery</h1>
<p>
  An ML system gains life when a need is discovered. By need, we distinguish
  between a request for a specific feature, such as a linter (a tool we use to
  ensure data quality), and the underlying operational or business pain point or
  problem - such as a new labelling capability requiring ML support. It’s not
  uncommon for operational or business teams to stumble into the
  <a
    href="https://en.wikipedia.org/wiki/XY_problem"
    rel="noopener noreferrer"
    target="_blank"
    >XY problem</a
  >
  by pattern matching and requesting a clone of an existing feature, without
  realizing that their underlying problem is actually quite different. Or, they
  might ask for something which is incredibly difficult, unaware that a slightly
  modified version would save literally months of development time. So an ML
  engineer should always hunt down the true problem that needs to be solved.
</p>
<p>
  Pain points and problems can be project-specific or exist horizontally across
  business teams or units. Historically, many of the problems that the ML team
  at Scale have tackled are <strong>project-specific</strong> and brought to us
  by our cross-functional stakeholders. For example, a project manager running
  into issues with a labeling project or a product manager looking to launch a
  new feature will surface these project-specific issues to our team. Tackling
  these isolated problems can result in marooned solutions which aren’t readily
  reusable.
</p>
<p>
  Solving <strong>horizontal issues</strong>, in comparison, maximizes our
  leverage, since the ML systems we build will then scale with our business.
  However, cross-business issues that can be solved with ML are often harder to
  identify because of the sheer operational complexity. These issues require a
  more active discovery process and take more time and deeper alignment with
  partners. The problems with the highest stakes are not obvious to ML
  engineers, who don’t get day to day exposure to them, even as these frictions
  impose significant pain on the operations or business side. To get a sense for
  this, imagine explaining the pain of managing multiple python environments and
  package versions to an operational lead - it can be incredibly painful for
  you, yet esoteric, even cryptic, to them.
</p>
<p>
  As another example, while working on an image semantic segmentation project,
  we learned that pre-labels generated by a model specific to that project were
  sometimes ignored by taskers, but the root causes were unclear. A close
  investigation identified a number of culprits, ranging from the inflexibility
  of pre-labels to having optimized for the wrong metric during development.
  <a
    href="https://scale.com/blog/autosegment"
    rel="noopener noreferrer"
    target="_blank"
    >Autosegment</a
  >, developed by Sean Li, Saleh Hamadeh, and the two of us, aims to tackle
  these problems. Autosegment solves a challenge for all image segmentation
  projects and isn’t restricted to a particular project. Learning about this
  challenge and building the appropriate solution required input from numerous
  stakeholders - from the project leads, who wanted to accelerate taskers; the
  taskers themselves, who used our tools and knew what worked and what didn’t;
  to engineers, who were familiar with the implementation of image segmentation
  in our interface and understood the kinds of tradeoffs that different tools
  would be need to make.
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7866cd69-bd81-40f7-4e44-6094a84f2c00/public"
    alt="Stakeholder concerns"
  />Example overlap of concerns between different stakeholders for a new
  ML-assisted tool.
</p>
<p>
  Identifying the pain points of the end user always requires talking to the end
  user. Instead of theorizing what could be a problem for them, it’s always more
  effective to speak to users directly and observe them as they encounter the
  problem. When building systems to accelerate labeling operations, ML engineers
  need to build user empathy. A simple way to build user empathy is to spend
  time tasking and dogfooding our own platform. There is no substitute for first
  hand experience for understanding what makes tasking hard.
</p>
<h1>Phase II: Design</h1>
<p>
  After Discovery, the project enters a <strong>design phase</strong> where an
  initial approach is sketched out - what kind of model to build, what sort of
  data is needed, the timelines, and so on.
</p>
<p>
  Designing effective ML systems at Scale requires two things: a clear need,
  which we’ve discussed above, and knowing the many different ways a model can
  fit into the labelling pipeline.
</p>
<p>
  This last point is worth further elaboration. The same machine learning model
  can be used for a variety of operational purposes. For example, a model that
  predicts bounding boxes for cars can be used to:
</p>
<ol>
  <li>Estimate the number of boxes in a task (task difficulty estimation)</li>
  <li>Prepopulate the scene with bounding boxes (pre-labelling)</li>
  <li>
    Suggest fixes to bounding boxes made in real-time by a tasker (active
    tooling)
  </li>
  <li>
    Identify missing boxes or unexpected boxes in a completed scene (linting)
    For each of these, the inference API could have the exact same request and
    return formats. However, each of the four approaches above require different
    operational adjustments, not all of which may be feasible. Task difficulty
    estimation makes assumptions about what makes a task difficult or easy for
    humans and implicitly maps these to legible subpopulations of taskers.
    Active tooling can require nontrivial operational investment in teaching
    taskers how to use the new tool and engineering lift to build new frontend
    components. Building a useful ML system means designing how an ML model will
    interface with the rest of the operational and engineering systems, in
    conjunction with actually training the ML model.
  </li>
</ol>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8e6d0b0b-5185-4a7c-227e-03350b1bdd00/public"
    alt="One model, many uses"
  />The same model may be applied in different ways to an existing pipeline,
  requiring different accommodations by engineering or operational teams.
</p>
<p>
  We’ve noticed two particular traps for ML engineers which can lead them on
  wild goose chases during the design phase. The first is misalignment between
  the machine learning problem and the business problem. The second is confusing
  the customer and the end user of a system.
</p>
<p>
  ML engineers are trained to use models to solve machine learning tasks, like
  box detection or text classification. For a box detection dataset, you’d use a
  box detection model. For a text classification dataset, a text classification
  model. When the business problem looks like an image classification problem,
  you might reach for an image classification model, only to discover that what
  you really need to solve is tabular regression.
</p>
<p>
  Faced with using ML to assist, say, labelling semantic segmentation training
  data, a common reflex is to train a semantic segmentation model to do the
  labelling. But if you already had a capable model, you wouldn’t need to
  collect training data for it! The mistake here is to apply the familiar task
  paradigm - semantic segmentation - to solve the business problem -
  accelerating semantic segmentation.
</p>
<p>
  Many operational questions take the form “Is something X or not X?” To an ML
  engineer, this sounds like a classification problem, and it’s easy to reach
  for a classification model which directly answers the question. Putting aside
  whether modeling X directly is feasible, it’s often the case that we already
  have an operational or engineering system which could answer this question
  more effectively with some additional information. In this situation, it may
  be faster to figure out how to extract the augmenting information rather than
  replicate the existing system altogether; a strong reason to apply a modeling
  scheme that looks very different from the modeling question.
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3505a43e-5c45-4274-cdd3-2e27242e0600/public"
    alt="Leveraging existing systems"
  />Modeling the business problem to solve doesn’t always mean reaching for the
  most obvious machine learning task paradigm.
</p>
<p>
  The second trap is mistaking the end user of a system with the people who
  request or manage the system. When we build a tool like
  <a
    href="https://scale.com/blog/autosegment"
    rel="noopener noreferrer"
    target="_blank"
    >Autosegment</a
  >, even though the end user of the active tooling is the Tasker, the internal
  customer is actually the Delivery Operations team. What we build has to meet
  the customer’s expectations, so even if Taskers enjoy using our tool, we need
  to make sure to move the needle for the Delivery Operations team. But the
  expectations and needs of the end user may differ or conflict from the
  internal customer (e.g. latency tradeoff with cost).
</p>
<p>
  In order to avoid falling into these two traps, ML engineers have to be
  deliberate in how they transport the business problem into a machine learning
  context; and always keep in mind the different stakeholders.
</p>
<h1>Phases III - VI: Decision and Alignment Through Continuous Improvement</h1>
<p>
  After the initial Discovery and Design phases, the project needs a go / no-go
  decision by both the
  <strong>ML team as well as cross-functional teams</strong>. In this phase, all
  teams must decide whether to allocate the necessary resources to the project.
  If the ML team decides to dedicate resources but the cross-functional team
  cannot commit the time or spare the bandwidth, the effort will ultimately fall
  short. Some factors teams must consider when making this decision include:
  prioritization, level of effort needed, and available bandwidth.
</p>
<p>
  Once all teams commit to the project, we can <strong>develop</strong> the
  ML-specific components of the ML system. This is when we build data pipelines,
  train models, and ship an endpoint. Once the model is developed, the project
  nearly always needs engineering and operational <strong>integration</strong>.
  For example, if we are developing a new pre-labeling model, the model requires
  integration with our labeling pipelines to attach the pre-label to the task.
</p>
<p>
  Now the project is actually <strong>deployed</strong>. Deployment should also
  follow best practices: meaning the feature should only be enabled for a small
  percent of tasks or users at the beginning, making it easier to revert if
  there are any issues. The system should then be constantly
  <strong>measured</strong>. Especially in the first few weeks, close attention
  should be paid to ensure no issues arise.
</p>
<p>
  We then use the metrics and qualitative feedback gathered in the measurement
  step to <strong>continuously improve</strong> the model. Models often need to
  be retrained to account for new edge cases. As we maintain the production
  model, we often encounter data distribution shifts or taxonomic changes which
  must be addressed.
</p>
<h1>Conclusion:</h1>
<p>
  We hope this post provided some insight on how we design our ML systems and
  how you and your teams can effectively partner with horizontal teams to deploy
  models that serve your business needs. Throughout the phases of ML
  development, we strive to ensure alignment with our cross-functional
  stakeholders. While gaining and maintaining alignment takes time and
  deliberate effort, it allows us to unlock capabilities on our platform that
  wouldn’t otherwise be possible.
</p>


Understanding operational and business context is critical to shipping ML systems that work. 

How we leverage concepts from design thinking to design ML systems in our internal processes to deliver results for customers.

Design Thinking for ML Systems at Scale

<p>
  It’s a story that you’ve heard before: the biggest and best machine learning
  models need enormous datasets to get good performance on complicated tasks.
  When you are just getting started with deep learning, the sheer volume of data
  that you hear the state-of-the-art models need can be pretty discouraging.
  However, today I’m going to show you how to stand on the shoulders of giants
  to supercharge your next ML project on a tough task like object detection. Do
  you want quality results without having to train on the same mass of data?
  Enter: transfer learning.
</p>
<p>
  (For those of you who are already familiar with transfer learning, feel free
  to skip ahead to the problem setup and results)
</p>
<p>
  “Transfer learning” refers to when we take a model that was trained on a
  related task and retrain it on a new task of interest. We can then piggyback
  off of the hundreds/thousands of computing hours spent training the existing
  model on some other task and train on a much smaller dataset and just
  fine-tune the model weights to be accurate for our use case.
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b82a3a9f-ac11-45c4-af64-005e9115d900/public"
    alt="Transfer Learning Diagram"
  />
</p>
<p>
  We effectively leverage all the expertise and hard work leading companies and
  research groups have already invested to create a good model then take out the
  last layer and swap it to match our application. Intuitively, since the model
  was trained on a related task, the weights and internal representations that
  it already learned should be useful for the new use case. All the normal model
  training tips and tricks apply, except for the fact that we initialize the
  model with the pre-trained weights.
  <strong
    >Transfer learning is a really important technique for making many
    production-ready ML systems work well.</strong
  >
</p>
<p>With the explanation out of the way, let’s see it in action!</p>
<h2>The Scenario</h2>
<p>
  Let’s say that I’m an ML-savvy rancher trying to improve the automated
  monitoring of my horses with a state-of-the-art object detection system (a
  setting I’m sure you’ve all encountered before). Some initial research
  suggests that there aren’t any models out there that are pre-trained on horse
  detection. Unlucky! Good thing we have Rapid and transfer learning.
</p>
<h2>Data</h2>
<p>
  For my data, I found an open source collection of animal images. With Rapid, I
  uploaded the images and instructed taskers to draw a tight bounding box around
  any horses present in the image. I then submitted a small calibration batch to
  make sure the labelers understood the task at hand and to spot check my own
  data. Once that was done, I was galloping along! I had 147 images labeled with
  bounding boxes around the horses back to me within 2 hours, all for around
  $20.
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d68acc6a-0a3c-499e-7f89-b29128955900/public"
    alt="Images of horses found in Scale Nucleus"
  />
</p>
<h2>Model</h2>
<p>
  My pre-trained model that I’m transferring on is the
  <a
    href="https://arxiv.org/abs/1506.01497"
    rel="noopener noreferrer"
    target="_blank"
    >Faster R-CNN</a
  >
  model with
  <a
    href="https://arxiv.org/abs/1512.03385"
    rel="noopener noreferrer"
    target="_blank"
    >ResNet 50</a
  >
  backbone.
</p>
<pre>
<code>def get_pretrained_model(): # load model pre-trained on COCO model =
  torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
  num_classes = 2 # 1 class (horse) + background input_features =
  model.roi_heads.box_predictor.cls_score.in_features # the key part: swap out
  the last layer for a predictor that matches our needs
  model.roi_heads.box_predictor = FastRCNNPredictor(input_features, num_classes)
  return model</code>
</pre>
<p>
  This model is already trained on the famous
  <a
    href="https://cocodataset.org/#home"
    rel="noopener noreferrer"
    target="_blank"
    >COCO (Common Objects in COntext) object detection dataset</a
  >.
</p>
<h2>Training</h2>
<p>
  Now that I have a model and data, I just need to train. After slicing my data
  as I desire in Nucleus, it’s easy to keep my images and files bundled together
  neatly:
</p>
<pre><code>import nucleus SLICE_ID = 'slc_c8c7418zpj7g0csycj5g' # a slice of my animal
  dataset client = nucleus.NucleusClient(API_KEY) data =
  client.get_slice(SLICE_ID) data_list = data.items_and_annotations()</code></pre>
<p>
  With all the data horseplay out of the way, we just have to load our model and
  data into the processor and let it do its thing.
</p>
<h2>Results</h2>
<p>
  I trained the model for 10 epochs, which took about 15 minutes on GPU (or
  around 4-5ish hours on CPU if you’re not pressed for time). We achieved an F1
  score of .517 (higher is better; when Faster R-CNN was initially released, its
  F1 score on COCO was .507). Way to go transfer learning! Here are some of the
  examples from the evaluation set:
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9b2da0d8-160b-490a-7146-8104f29d6500/public"
    alt="Examples from the evaluation set "
  />
</p>
<p>
  Not bad, but we could do better. At this point I could spend more time tuning
  hyperparameters or training for more epochs. However, I’m too busy running the
  ranch to spend time doing hyperparameter sweeps or ensembling models.
</p>
<p>
  Normally a bottleneck for model improvement is a lack of quality labeled data.
  Thankfully with Rapid, there’s no such bottleneck. I still have unlabeled
  images in my dataset – let’s see what happens when I change nothing else but
  add more labeled data!
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/a275634f-e792-4be1-469d-c9ab175fe800/public"
    alt="Additional images labeled in Rapid"
  />
</p>
<p>
  The newly labeled data brings my total to 351 annotated images, which only
  cost me around $45 on Rapid. After 10 more epochs with the new data
  incorporated, the model improves our F1 score to .655.
</p>
<p>
  Much better, and I didn’t even need to do any hyperparameter tuning! Let’s
  compare the new and old model predictions:
</p>
<p>
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e29a9815-ef1f-45f6-f82d-260121577700/public"
    alt="Examples from the evaluation set"
  />
</p>
<p>
  With more data, it seems that the model gets a little less confused with
  crowds of horses and captures more of the visible edges of the horse.
</p>
<h2>Conclusion</h2>
<p>
  We’ve seen how to apply transfer learning to a custom use case. Now all I need
  to do is productionize and deploy my custom horse detection model, and I’ll be
  sure to take my ranching (is that a word?) to the next level. I hope you’ve
  been inspired to try out Rapid for your next transfer learning project! You
  can
  <a
    href="https://dashboard.scale.com/login?redirect_url=rapid"
    rel="noopener noreferrer"
    target="_blank"
    >get started with Scale Rapid today here</a
  >. Want to get a jump start for your own experiments? Access the code from my
  experiment on
  <a
    href="https://github.com/scaleapi/dev-blog/tree/main/nucleus-rapid-blog/transfer_learning"
    rel="noopener noreferrer"
    target="_blank"
    >Scale's Github</a
  >.
</p>


In our latest blog, we explore how to implement transfer learning using Scale Rapid.

How to Implement Transfer Learning for Object Detection with Scale Rapid

Easy Transfer Learning with Scale Rapid

<p>As a member of the U.S. Air Force Rescue and #AFWERX, I have been working to modernize the tools used by Search and Rescue to increase the odds of survival of those lost at sea. Every year, hundreds of individuals find themselves isolated and at the mercy of the ocean, creating a dangerous scenario which is impossible to navigate without the best technological tools and deep expertise. </p>
<p>Being in the ocean is one of the more extreme environments to be isolated. The conditions that make it hard to survive also make it difficult to locate and rescue the isolated; rescue is rarely a simple mission. Attempting to modernize the Search and Rescue model is arguably just as difficult. </p>
<p>Systems are out of date: the “how” hasn’t changed in over 20 years. And changes that have been adopted, have not been implemented at the speed of relevancy. </p>
<p>Five years ago, I began exploring different approaches to solving Search and Rescue’s toughest problem statements. During this discovery process, two significant experiences led me to become a Fellow at Scale. </p>
<p>In my first experience, I was deployed as a United States Special Operations Command  (SOCOM) Tech Advisor as a Personnel Recovery Subject Matter Expert (SME). I had the opportunity to share these problem statements with Josh Wolfe, co-founder of Lux Capital, and the rest of the SOCOM Tech Advisory Panel team. Josh shared his opinion on how the DoD could use a page out of the venture capital playbook: to move faster with sustainable speed in developing solutions. </p>
<p>Seeing how Josh was investing in startups that invent the future inspired me. I began to engage with early-stage startups to find new solutions that would save lives. </p>
<p>The second instance was at a Hacking for Defense session at Stanford University. This time, I was working with young college students learning the lean startup methodology to solve DoD problems. Steve Newell, the founder and CEO of BMNT, introduced the Search and Rescue problem statement to the class. He presented it in such an elegant way—I could physically see his inspirational words produce passion on the young data scientists’ faces. </p>
<p>At this moment, I realized that AI developed to save lives would be the catalyst for large-scale DoD adoption. </p>
<p>A few years later, and several AFVentures Small business Innovation Research contracts later, Forrest Underwood introduced me to Shift.org. This was the opportunity I couldn't pass up: the chance to work inside a startup for 8 weeks. </p>
<p>Shift was able to recognize a critical flaw in my strategy to solve problems. How can you get the DoD to adopt industry AI solutions without possessing a well-developed ML pipeline? For this reason, I think Shift matched me with Scale AI.</p>
<p>Scale AI is the best-in-class at providing the entire ML lifestyle to its customers so they can focus on the real, pressing problems at hand. This is exactly what's needed when a DoD customer asks for an AI solution. </p>
<p>It is a misunderstanding of the requirements surrounding ML that can be difficult when attempting field AI solutions. Without a mature ML pipeline, it’s very difficult to have your model perform correctly and continue to improve as the extremely dynamic world evolves. </p>
<p>On the first day on the job, I attended #TransformX, a virtual conference with the CEO Alex Wang, a community of leaders, visionaries, practitioners, and researchers across industries. This was probably one of the best onboarding or familiarization phases that I've experienced in 20 years.</p>
<p>Spending 2 days listening in on thought leaders discussing the art of what's possible in a long forum was extremely insightful. </p>
<p>Throughout the entire fellowship experience, I compared how Scale operates against Air Force Institutions of which I’ve been a part. One observation that stood out to me was the holistically flat organizational rank structure of Scale. After seeing the level of engagement that Alex Wang had during TransformX and the scope of his responsibilities, it was easy to draw parallels between Alex and a Geographic Combatant Commander. </p>
<p>One welcoming attribute of Scale was the inclusion of ambitious ideas. It is understood that the company is at the edge of the AI frontier, and new ideas are required to move forward quickly. This is what has allowed Scale to grow at an incredible rate; it’s also the attribute that the Air Force is trying to re-install into its ranks. </p>
<p>Inside Scale, I had the opportunity to work for the Federal division. The level of talent and prior experience of the team was truly awe-inspiring. I had a false assumption that I would be working with individuals without experience in the Armed Forces. I was surprised when I learned that there was a rich mix of individuals from both the DoD and Silicon Valley. </p>
<p>At first, it felt like I walked into a Special Operations Group with Senior Instructor Pilots, Chief of Intel, and the Ground Forces Commander discussing an Operations Order. Except, everyone was in business casual and there were lead engineers in the conversation. </p>
<p>For Scale to recognize the talent pool coming out of the DoD was exciting to see. The method of mixing operators, strategists, and engineers inside is not common practice for the Air Force. The standard Air Force approach is to silo these different functions on a day-to-day basis.</p>
<p>Even my two cents were considered as the new guy. That says a lot about an organization. </p>
<p>The art of creating what's possible is something that Scale was always discussing with potential customers. This wasn't a "fake it until you make it” model. The conversation was followed up with things that were deployed in that space to demonstrate their success. My gained understanding of what's possible is what I will take back to the Air Force. Thank you Scale.</p> 

Government

Bon Strout details his experience as a Scale Fellow

Scale Fellow Spotlight on Bon Strout

Shift Ventures Fellow Spotlight: Bon Strout

<p>When my co-founder and I started Plaid in 2012, the financial system was on the precipice of a fundamental restructuring. At Plaid, we provided infrastructure to enable developers to build novel financial applications that we never thought possible just years before. Whether it was in payments, underwriting, investing or whatever a developer could build, we enabled tens of thousands of developers to catalyze an industry that now empowers hundreds of millions of consumers and SMBs to take control of their financial lives.</p>
<p>As I watched Scale grow, it became clear that Scale in many ways was in a similar position to Plaid—it had become the underlying infrastructure player to an industry at an inflection point. </p>
<p>AI/ML has the fundamental ability to restructure the way that systems and businesses are built. Yet because of the vast amounts of unstructured data required, practical implementations of AI have been frustratingly limited. Scale stands alone in their focus and ability to change that. Alex and the team at Scale understand that model development alone is just one part of the complex AI supply chain. By enabling others to leverage the most advanced data annotation system and model supply chain, they’ve rewritten the narrative and enabled any company and industry to rethink how their systems are built and operated. Whether it's powering Toyota to build the future of automotive, helping Flexport improve our supply chains, or enabling the Department of Defense to better protect its citizens, Scale has the ability to dramatically amplify the real world impact of AI and ML. </p>
<p>As one of their credos says, <em>ambition shapes reality</em>. There is no team or company that is better positioned to enable AI to shape the world around us than Scale. I could not be more excited to join Alex and the rest of the board to help build out and accelerate an already incredible team.</p> 

Scale board member William Hockey details why he believes AI/ML has the fundamental ability to restructure the way that systems and businesses are built.

Scale welcomes William Hockey as a board member

<h1 id="update">Update:</h1>
<h1 id=""> </h1>
<p>We have received an overwhelming response to our announcement of free, AI-ready datasets in support of Ukraine. Nearly a hundred AI companies, researchers, developers, and GIS practitioners have offered to support the initiative with Scale – an inspiring global community of AI Developers for good.</p>
<p>Given the high demand, we will provide the datasets in phases starting with requestors with the greatest need, followed by a broader community. At this time, our focus is on providing Structure Damage Assessments in Kyiv, Kharkiv, and Dnipro.</p>
<p>The path forward is now two-pronged – our top priority is the delivery of damage assessments to those with urgent need. </p>
<p>The second prong of our effort is to provide AI-ready training datasets for Damage Assessment to the broader AI community, starting with a structure recognition training dataset. </p>
<p>Today, we are sharing an initial dataset with the broader AI community: <a href="https://drive.google.com/uc?id=17lpMFngiqeegIm1U0RzUYfB-9GjmI7fk&export=download">link to download dataset</a>.</p>
<p>Note: the corresponding ZIP file's SHA256 Checksum is: 379c394dab6a4d7f1358625d086ebea4b39aaa124548e8c079bee16fb574bc53.</p>
<p>We appreciate your patience as we assess and prioritize requests for data and data development. We look forward to continuing to support Ukraine alongside other AI Developers leveraging critically important open-source and commercially available data for this important work. </p>
<p>Sincerely,
Scale AI Team</p>
<p>———————————————————————————————————————————</p>
<p><em>At Scale, we believe in using AI technology in support of democratic values. That’s why starting today, Scale will be providing a series of AI-Ready datasets that algorithm developers can use to rapidly train and deploy AI in support of Ukrainian operations. By providing these datasets at no cost to national security practitioners, we hope to support a diplomatic solution and swift end to this conflict.</em></p>
<p><em>Diplomacy is fueled by insights; foundational to that is quality, trusted data. We have recently created strategic partnerships with commercial satellite imagery providers in order to develop these datasets, which include Aircraft Detections, Structure Damage Assessments, Change Detection, and the JSON and GeoJSON annotations. We encourage AI Developers to contact detections@scale.com to receive this data free of charge.</em></p>
<p>To better understand how these datasets can be used, Scale presents a contemporary use case of our partnerships with national security leaders. We deliver an AI-powered change detection solution capable of identifying and tracking military formations across the globe. </p>
<p>The video below demonstrates continuous monitoring and improvements by Scale’s ML engineers and deployment strategists. We provide timely and accurate insights through our AI Data Readiness platform, our Test and Evaluation platform and our continuous ML Ops tooling. </p>
<p>Leveraging Scale's AI/ML ops platform, we continuously develop and hone deep learning models to describe scenes and detect objects across a range of inputs from commercial satellite imagery providers.</p> <div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/hphfibwbn6" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div><div class="markdown-module-blog"><p>We are now delivering insights to U.S. national security customers from this full-stack, multi-intelligence, human-teaming product, including on Russian formations across Europe and Eurasia. </p>
<p>Here, on Russian air and ground order of battle (OOB) data, we successively identified OOB changes across the EUCOM area of responsibility (AOR) over the past several months. We used high-resolution synthetic aperture RADAR(SAR) imagery from Capella Space and high-resolution optical imagery to deliver high-accuracy, time-sensitive insights.</p>
<p>“In these intricate and time-sensitive scenarios, it is critical that we deliver insights from multiple, disparate datasets at the speed of relevance,” said Mark Valentine, Head of Federal at Scale. “We are proud to demonstrate the power of our platform, and our approach to human-machine teaming to support national security leaders and practitioners.”</p>
<p>Monitoring the Russian OOB on the border of Ukraine using commercial satellite imagery, and other non-traditional Intelligence, Surveillance and Reconnaissance (ISR) sources requires adaptation to a variety of challenges in a constantly-evolving field. </p>
<p>The challenges in using AI to analyze OOB changes include: exigent scenarios presented by weather, disparate image varieties and quality, and the inclusion of relatively rare military objects such as transporter erector launchers (TELs), for which there are limited real-world observations.</p>
<p>The depth of ontology can also present complications: the presence of multi-generational aircraft, ground vehicles, RADAR and TELs mean that models need to be trained against tens of thousands of observations in a range of environmental conditions.</p>
<p>If you are an AI Developer who would like to receive this data free of charge, please contact detections@scale.com.</p></div>

Scale AI provides free datasets for national security partners

Scale AI providing free datasets to help national security partners gain current insights into Russia-Ukraine conflict

<p>Scale Rapid is a quick and easy way to get clean, high-quality labeled data for your next ML project. We used it to train a <a href="https://scale.com/blog/imdb-rapid">sentiment analysis model</a> in a previous post  - and we’re back now to show you how to use Scale Rapid to train an unsupervised model!</p>
<p>In this blog, we are going to use <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix">CycleGAN</a> (<a href="https://arxiv.org/pdf/1703.10593.pdf">paper</a>) to anime-fy images with Scale Rapid. </p>
<p>CycleGAN is trained in an unsupervised manner, meaning each X input does not need to correspond 1:1 to a y output in order to train the net. </p>
<p>Instead of using Scale Rapid to label training data, we used it to clean data. We separated a group of images scraped from the internet into categories. This classification allows us to make the two inputs necessary for CycleGAN - a class A (real-life images) and class B (Japanese anime-style images).</p>
<p>To start, we downloaded 10k images from Google images from a variety of different searches. At a glance, the corpus had all sorts of images.</p> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/27b06dda-1661-4805-c721-0ec70ee20300/public" alt="Sample images from Google" />
        </figure><div class="markdown-module-blog"><p>Class A contains real-life images and Class B contains anime-style images. Image 3 belongs to Class A, and Image 2 to Class B. Images 1 and 4 do not belong in the dataset as they are neither real-life nor anime-style images.</p>
<p>The ask for Scale Rapid is to discriminate between these three types of images: a real-world, realistically edited photograph, a Japanese-style anime scene, or neither. Rapid responses are then used as ground truth to establish which images should and shouldn’t be in the dataset we train CycleGAN on.</p>
<p>Rapid’s categorization endpoint supports image categorization - and it's only 8 cents per image!</p>
<p>Because CycleGAN is trained on <em>unsupervised</em> data, we don’t need to pair class A and class B images - we can simply separate them from one another and train.</p>
<h2 id="usingscalerapid">Using Scale Rapid</h2>
<p>Scale Rapid works in two phases- first, the user sets up the project &amp; instructions, and runs small calibration batches to ensure the initial results are as expected. Once quality on those looks good, larger production batches may be uploaded for labeling.</p>
<p>For this application, we set up a categorization project. The taxonomy is just one multiple-choice question:</p>
<p><strong>What type of image is this?</strong></p>
<ul>
<li>Anime (japanese animation) image</li>
<li>Photorealistic (real-life) image</li>
<li>Neither</li>
</ul>
<p>Scale Rapid also has a feature called long hints, which allows us to prompt labelers with specific things to think about upon choice selection via a tooltip. For the first choice (Anime (japanese animation) image) I clarified with the following long hint:</p>
<p>"Japanese animation refers specifically to anime, as in the instructions. Other cartoon-style or generated images belong in the neither category."</p>
<p>We begin by uploading a calibration batch, and follow up with production batches. The labeling results are easily parseable via JSON. </p>
<h2 id="scalerapidresults">Scale Rapid Results:</h2>
<p>A random audit of tasks found just one mistake out of 200 randomly selected labeled images. A few examples of correctly labeled images are below, organized by class.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d6e522b3-b00e-4e91-47f5-3f709d144300/public" alt="Examples of correctly labeled images by class" />
        </figure><div class="markdown-module-blog"><p>With the help of Scale Rapid, raw data from the web was transformed into cleaned and confirmed data and separated into class A, class B, and neither. </p>
<h2 id="model">Model</h2>
<p>CycleGAN produces mappings from class A to class B and vice versa using two different types of losses- adversarial loss and cycle consistency loss. </p>
<p>Adversarial loss incentivizes the mapping to generate images that look similar to the target set. We achieve this by optimizing against an adversary D–a discriminator that tries to separate the generated images for the target class from actual data from the target class.</p>
<p>Cycle consistency loss incentivizes the mapping to produce images that look like they fit the distribution of the target class.This occurs by minimizing the difference between original data and that same data transformed into the same target class.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/cdcd9177-a511-4863-70fa-6a9c1194dd00/public" alt="Cycle consistency loss" />
        <figcaption>Cycle consistency loss is calculated by the difference between the untransformed class A image and the double-transformed class A image. The mapping aims to minimize this.</figcaption></figure><div class="markdown-module-blog"><p>We trained CycleGAN with our Scale Rapid sorted dataset.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3c9b78d0-1aaa-4d49-ad1d-e7d91e37ca00/public" alt="Loss Function Graph" />
        </figure><div class="markdown-module-blog"><h2 id="modelresults">Model Results:</h2>
<p>We are now able to anime-fy images! A few examples:</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5a438327-e4b7-41e5-a19b-2cd457c94d00/public" alt="CycleGAN Images" />
        </figure><div class="markdown-module-blog"><p><a href="https://scale.com/rapid">Try Scale Rapid</a> for your own project today!</p></div>

In this blog, we explore combining Scale Rapid Labeling with CycleGAN to anime-fy images.

Using Scale Rapid to train an unsupervised model.

Using CycleGAN to Train Unsupervised ML Models

Scale Nucleus

Transportation

Using Rapid and Nucleus together will get you better results than using Rapid alone.

 Using Scale Nucleus with Rapid to Unearth Errors—Label Smart, not Hard

Peer-Reviewed Doesn’t Mean Perfect Data

<p>Mission critical industries including logistics, financial services, government, and healthcare rely on high quality document data extraction in order to optimize business workflows, save money, and improve customer experience. In this article, we explain the conventional approaches for document processing, deep dive into drawbacks with these solutions, and introduce a new approach that enables higher accuracy, even across the most challenging document types.</p>
<h3 id="whatisocr">What is OCR?</h3>
<p>Optical character recognition (OCR) is a class of document processing technologies that, as the name suggests, reads characters from text on a page. Originating in the 1920s, OCR has since evolved to support use cases involving data extraction from digitized documents. There are two kinds of OCR technologies:</p>
<p>First, generic OCR, which pulls all of the text on a digitized document (image, PDF, etc.) and outputs a raw list of the words on a page. With a number of improvements in character recognition technology, generic OCR has become suitable for distinguishing words and reading complex font types.</p>
<p><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/98229f9a-1e6c-4a80-7416-5a38dfbd1d00/public" alt="Generic OCR: extraction/digitization but no field classification" /></p>
<p>Second, intelligent OCR, which has become a popular alternative since these systems can automate data extraction for key fields of interest rather than just receiving a blob of all the text on a page. Intelligent OCR is built on generic OCR, but enables a user to define a specific template for a document, manually specify all the fields they need extracted, and automatically get only the structured data they need for all similar-looking documents.</p>
<h3 id="problemswithocr">Problems with OCR</h3>
<p>There are clear challenges with both of these conventional OCR approaches that reduce the quality of data extraction.</p>
<p>Generic OCR simply outputs all the text on a page and doesn’t extract only the key information you might need for downstream workflows. These solutions are typically paired with complex regex or hard-coded heuristics that could require a team of nearly 4-5 engineers just to support parsing out required fields and maintaining the infrastructure. Even then, the variability in text layout and document format ultimately results in poor data extraction accuracy (~50-70% for simple documents, 10%-60% for complex unstructured documents).</p>
<p>Intelligent OCR can extract fields you need in a document, but requires manually creating templates and rules to define the location of those fields for every document layout. This approach can quickly become  painstaking and incredibly difficult to scale as organizations expand use cases. These solutions also fail when it doesn’t find a matching template on new layouts or when you need to add new fields to extract—common in real-world scenarios that have high volumes or complex, unstructured document types. While intelligent OCR approaches might get up to 70-80% data extraction accuracy with heuristics, it still requires significant engineering and manual operations support to ensure errors don’t fall through the cracks.</p>
<p><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6eb13fad-3577-422f-8cc9-51fc0a1eb100/public" alt="Intelligent OCR: extraction and digitization, classification from templates only" /></p>
<h3 id="tosummarize">To summarize:</h3>
<ul>
<li>Generic OCR simply extracts text but doesn’t identify the specific fields you need extracted. This approach requires significant engineering effort to maintain the infrastructure and yields low accuracy data.</li>
<li>Intelligent OCR can extract your desired fields but requires manually defining templates and hard-coded rules. This approach can get to higher accuracy levels than generic OCR but still results in many errors and doesn’t scale to large volumes and high variability use cases. </li>
</ul>
<h3 id="whyuseadaptivemldocumentprocessinginstead">Why use Adaptive ML Document Processing instead?</h3>
<p>Running OCR is the first of many steps needed to have a robust document processing solution that can extract data at the highest quality. While these solutions may get you part of the way there, they don’t deliver a complete solution because OCR still requires significant engineering and operations support in order to pull out key fields and fix errors. </p>
<p>While we hear about machine learning frequently in moonshot applications like autonomous driving and Artificial General Intelligence, the technology can be significantly useful today in enterprise use cases. Leveraging machine learning is key to getting a high quality and truly automated document processing solution even across highly variable and challenging document types. Rather than relying on off-the-shelf methods or ones that need manually defined templates, an end-to-end machine learning based approach can enable a robust solution with higher accuracy results for highly variable unstructured and structured documents.</p>
<p><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8d041be1-a4c2-4e5b-7641-785b4c115000/public" alt="Scale Document AI Adaptive Document Processing: the Best of All Worlds" /></p>
<p>However, setting up these systems in-house requires significant investment across the machine learning stack in order to deploy a production system, including data labeling, model training, and continuous monitoring. Choosing the right partner to do this at scale can help you get higher accuracy, lower latency, and save time and money to spend on your company’s core competencies.</p>
<h3 id="whyscale">Why Scale?</h3>
<p>Scale Document AI supports customers with quickly deploying a complete document processing solution with guaranteed quality SLAs for any of their document types. We use our own in-house OCR engine for identifying the text on a page, but don’t stop there. What’s unique about our approach is that we leverage base models built on state of the art Computer Vision and Natural Language Processing research. These models are trained on millions of data points and we further refine them for each customer use case. This fine-tuning uses Scale’s proprietary data labeling and ML training infrastructure that we’ve developed over the past few years to deliver a solution that consistently performs at high accuracy – even across unique edge cases, customer requirements, or variable document types.</p>
<p>Scale provides Adaptive ML document processing for many enterprises and organizations across financial services, logistics, manufacturing, healthcare, real estate and government. For example, Scale delivers fast extraction from messy invoices <a href="http://scale.com/customers/brex">for Brex</a>, accurate and timely data from complicated shipping paperwork <a href="http://scale.com/customers/flexport">for Flexport</a>, and essential data out of titles, mortgage applications, and deeds <a href="https://scale.com/customers/states-title">for Doma</a> in real estate transaction processing.</p>
<p>If you’re looking for a solution that’s higher quality, lower latency, and free of the hassle of creating and maintaining templates, <a href="https://scale.com/document-ai">learn more and contact us</a>.</p> 

OCR and even Intelligent Document Processing fall short, while AI-based extraction delivers.

With adaptive ML models performing extraction, Scale Document AI goes far beyond OCR.

Comparing Scale Document AI to Traditional OCR

<p>Today, we are thrilled to announce the <strong>Early Access Availability</strong> of <a href="https://scale.com/synthetic">Scale Synthetic</a>, the most efficient way to enhance ML performance with data that complements real-world datasets. With Scale Synthetic, ML teams can combine the benefits of real-world data with diverse and realistic synthetic data to achieve desired data distributions. </p>
<p>Real-world data remains indispensable to developing performant models. ML teams, however, have encountered several challenges with relying on real-world data alone. Data privacy issues can prevent collected data from being used, collected datasets also inherit the biases of conditions under which data was collected, and collecting enough examples of edge cases and rare scenarios can be both costly and time-consuming. </p>
<p>With Scale Synthetic, ML teams can: </p>
<ol>
<li><strong>Overcome the challenges of real-world datasets</strong> while maximizing the return on investment (ROI) of real-world data by realistically augmenting real-world data or generating fully synthetic data from scratch.</li>
<li><strong>Bridge the sim-to-real reality gap</strong> by leveraging Scale’s global, human-in-the-loop network that ensures objects in augmented or fully synthetic scenes are realistically placed, and follow a logical path.</li>
<li><strong>Seamlessly integrate</strong> with Scale’s full ecosystem of products for near-zero friction to add augmented or fully synthetic data into existing ML data pipelines. </li>
</ol>
<p>Scale Synthetic currently supports the augmentation or generation of image, video, or 3D point cloud data but custom requests are also encouraged. </p>
<p>To date, Scale Synthetic is trusted by leading ML teams such as Kodiak Robotics as well as the U.S. Department of Defense to deliver high-quality synthetic data. Additionally, Scale Synthetic has been recognized in the 2022 Gartner® Innovation Insight for Synthetic Data* report as a Representative Provider in Sample Synthetic Data.</p> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/13cf537b-0b89-4b55-885f-069505c8c900/public" alt="Kodiak Robotics Quote" />
        </figure><div class="markdown-module-blog"><p>To continue to enhance and build on the foundations of Scale Synthetic, we are also pleased to welcome Joel Kronander as the new Head of Synthetic Data and Vivek Raju Muppalla as the Director of Synthetic Services. Joel joins Scale from Nines, a startup that applied ML and AI expertise to the field of medicine where he served as the Head of Machine Learning. Vivek joins Scale from Unity Technologies, where he was the Director of Engineering, AI and Simulation.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9281bfec-9ad9-49d8-d658-c16381bb0c00/public" alt="Joel Kronander and Vivek Muppalla Quotes" />
        </figure><div class="markdown-module-blog"><p>If you would like to get early access for Scale Synthetic, <a href="https://scale.com/synthetic">contact us today</a>.</p>
<p>*Gartner, “Innovation Insight for Synthetic Data”, Arun Chandrasekaran, Alexander Linden, Anthony Mullen, Farhan Choudhary, February 7, 2022.</p>
<p>Disclaimer- GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.</p></div>

Scale Synthetic

Autonomous Vehicles

AR/VR

Scale Synthetic is the most efficient way to enhance ML performance with synthetic data that complements real-world datasets.

Complementing Real-World Datasets with Diverse and Realistic Synthetic Data

Introducing: Scale Synthetic - Now Available for Early Access 

<h1 id="fieldengineeringatscale">Field Engineering at Scale</h1>
<p>Scale’s Field Engineering is a global team of creative engineers who collaborate with clients to understand their challenges and architect solutions. Our solutions are rooted in our technical skill, product expertise, and a thorough understanding of our customers’ complex needs. Whether the answer is a bespoke solution to solve a unique problem, or a new developer tool that can scale across Scale, everything we do aims to ensure our customers benefit from the full potential of Scale products and expertise.</p>
<p>Our work is at the intersection of customer success, product, and engineering, helping Scale’s customers succeed and reach their business goals with Scale  by unblocking them technically. Field Engineers are subject matter experts who know all of the products and technologies Scale has inside and out, and work closely with product managers to influence new product features.</p> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2893973b-04ee-4e9e-9c7e-6304d5b38a00/public" alt="Field Engineering Map" />
        </figure><div class="markdown-module-blog"><h1 id="adayinthelife">A Day in the Life</h1>
<p>Field Engineers start each day with a team standup. During the standup, we align on daily priorities and check in on our longer-term projects. There’s almost always an urgent customer issue to work on, but when we have downtime, we tackle longer term projects that will help the company move faster in the future. We’ll go over some of these longer term projects further down. One of the more fun parts of our daily standup is ending with the “Word of the Day”. As half of our team is located in Buenos Aires, Argentina, each day someone will share a word that they would like to know how to say in English or Spanish and we’ll share our own take, the different definitions, and any fun stories that go along with it. </p>
<p>After the standup, each day can look pretty different. Our days will often include at least a few customer meetings, where we will consult with our existing customers (think: the world’s leading ML companies) on technical aspects of their projects, any challenges they may be facing, and new initiatives coming up. Field Engineering is the customer’s technical contact to ensure all goes well with their work.</p>
<p>Covid-permitting, we use lunchtime to socialize with colleagues in the office while we eat all the amazing food that is provided for us. It’s a nice time to decompress and connect.</p>
<p>After customer meetings and any urgent issues have been addressed, we focus on our longer-term projects. We’re focused on making the lives of our customers, our internal partners, and ourselves as streamlined and scalable as we can. So, we spend significant time on internal tools, customer SDks, documentation and anything else we can think of to improve Scale’s developer experience.</p>
<h1 id="fieldengineeringinitiatives">Field Engineering Initiatives</h1>
<h2 id="introducinganewandimprovedscalepythonsdk">Introducing a new and improved Scale Python SDK</h2>
<p>2021 saw the introduction of a new version of our Python SDK. Rebuilt from the ground up to deliver new features faster, this new version includes a set of new ways to interact with the Scale API and improve infrastructure for a smoother developer experience. The tool was built on a new foundation that includes novel abstractions, so you can get more out of the Scale API with less effort. This new foundation allows us to add new functionality faster and better than ever before, so expect more new features from Scale to show up in the SDK with improved documentation and release notes.</p>
<h2 id="shippingalidartoolkittomakesenseofmultisensordatasets">Shipping a Lidar Toolkit to make sense of multi-sensor datasets.</h2>
<p>The act of fusing multiple cameras, lidar sensors, and radars from a vehicle or robot into one cohesive scene that can be annotated is no easy feat.</p>
<p>Historically, helping customers work through the nuances of their data and how to format it for Scale’s API-based workflows was one of the most time-consuming parts of our team’s day. In 2021 we overhauled our Lidar Toolkit and introduced a set of resources, code samples, and debugging tools to enable maximum efficiency for developers working with Lidar data.</p>
<p>The toolkit, built on Open3d and Scale’s Python SDK, is full of helper functions that make working with the geometry, camera projections, and coordinate systems much easier to understand and implement than doing it from scratch.</p>
<p>The Lidar Toolkit our team has published is now the de facto way all customers share and upload their data to Scale today.</p>
<h2 id="embracingtheoperationalnuancesatscale">Embracing the Operational Nuances at Scale</h2>
<p>Scale is in the business of human-in-the-loop data annotation, and humans + loads of data means a lot of operational overhead comes with the territory. </p>
<p>We don’t want to rely on running our business based on custom scripts day in and day out. So, we continually develop, iterate and ship internal tools to make the lives of our customer delivery team as seamless as possible. Some of the tools make facilitating customer requests easier, like moving tasks (tasks are what we call the unit of work we are annotating, like an image) between customer projects, re-sending callbacks to our customer’s endpoints, or canceling tasks en masse. </p>
<p>Other sets of tools ensure high-quality data is coming out the other side of our task pipelines. A few years ago, had we wanted to programmatically identify potential annotation errors, a custom script would need to be written, tailored to that client’s specific projects. Over the past 18 months, Field Engineering has developed a framework that can be configured by our non-technical ops counterparts to flag any erroneous tasks in their pipelines. And, better yet, this framework can proactively prevent these tasks by flagging warnings directly in the labeling interface—before the annotations are even submitted. This has become one of our most powerful tools to ensure reasonable data is consistently delivered to our clients.</p>
<h1 id="couldfieldengineeringbeafitforyou">Could Field Engineering be a fit for you?</h1>
<p>If you enjoy leveraging both sides of your brain, the people skills side, and the deeply technical one, Field Engineering may be a promising place to build and Scale your career. We are consistently hiring and growing the team (the team has doubled in size each year) and are interested in candidates with a variety of technical backgrounds and skills. Bonus points are given to candidates who possess a background in Computer Vision or API building and troubleshooting. If you're interested, <a href="https://scale.com/careers?location=All&team=Engineering">explore our open positions today</a>!</p></div>

A Day in the Life of a Field Engineer at Scale

Field Engineering at Scale

Field Engineering at Scale | A Day in the Life

<p><strong>In an exciting announcement to close out the year, Scale has been featured in two of Gartner, Inc.’s 2021 Hype Cycles.</strong></p>
<p><strong>We are honored to have been featured in Hype Cycles for both Artificial Intelligence and Data Science and Machine Learning.</strong></p>
<p>These reports detail emerging technologies in AI, data science and machine learning, and name Scale as a representative vendor for Data Labeling and Annotation Services.</p>
<p>The reports describe an urgency for data labeling and annotation services (DLA) to become widely adopted. According to Gartner analyst Anthony Mullen’s analysis, the demand for labeled and annotated data has risen in order to remove bottlenecks in industry-specific use cases.</p>
<p>And, due to the fact that most companies do not have in-house skills and processes for DLA, companies like Scale are the best option to provide DLA to yield optimal AI results.</p>
<p><strong>Why good data matters</strong></p>
<p>Better data leads to more performant models, and performant models lead to faster deployment. With better data, Scale helps customers deliver more value.</p>
<p>An increased diversity of use cases in DLA will drive both AI and industry-specific innovation, according to Gartner analyst Anthony Mullen. Vendors in the market have specific offerings across industries including commerce, robotics, autonomous vehicles, retail, GiS/maps, AR/VR, agriculture, finance, transportation and more.</p>
<p>Scale is proud to accelerate innovation for customers including PayPal, iRobot, Square, Brex, Pinterest, Etsy, SAP, Airbnb, Instacart, NVIDIA, OpenAI, Samsung, Toyota, GM, the U.S. Air Force, and more. </p>
<p><strong>How does Scale solve the bad data problem?</strong> </p>
<p>Scale is your AI readiness partner. We got our start in data labeling and annotation for autonomous vehicles. Now, Scale provides a data-centric, end-to-end solution to manage the entire ML lifecycle.</p>
<p>That means that Scale can meet you wherever you are in your data process. </p>
<p><strong>ML-powered, human-in-the loop data</strong></p>
<p>Scale’s strength is the combined power of our ML platform and expert human insights to help our customers develop their data strategy with high quality data from the ground up. </p>
<p>According to Mullen, there are AI solutions which would be impossible without human-in-the-loop labeling, including in cases of search engine tuning and image recognition.</p>
<p>Furthermore, DLA allows data scientists to allocate and prioritize time for higher-order tasks, instead of taking time and effort to annotate data themselves. </p>
<p>With Scale Rapid, any machine learning developer can achieve production-quality labels with no data minimums.</p>
<p>Receive quality labels in just a matter of hours, with detailed insights and identified edge cases. Scale Rapid can even handle subjective tasks with the utmost precision and quality.</p>
<p><strong>But what comes after labeled data?</strong></p>
<p>Gartner Inc., has made the case for why data matters to AI development. </p>
<p>After data labeling, what’s next? Data management. </p>
<p>Whether you have prepared the data, or we label and annotate the data for you, we’re your premiere data management partner. </p>
<p>Scale Nucleus is the command and control for your ML dataset. Visualize, curate and debug your dataset to improve your ML model performance with Nucleus.</p>
<p><strong>Scale’s advantage: AI development, from data labeling, through the entire ML lifecycle.</strong></p>
<p>Scale’s mission is to accelerate the development of AI. We do this by providing a data-centric, end-to-end solution to annotate, manage, automate, evaluate, collect and generate your data. </p>
<p>Our products and services meet you wherever you are in the ML lifecycle, whether you need data labeled and annotated, or need an ML engineer to help explore and refine how AI can help your business.</p> 

Gartner Hype Cycles for AI and Data Science and Machine Learning

Scale highlighted in Gartner Inc.’s 2021 AI Hype Cycles

Scale Highlighted in Gartner Inc.’s 2021 AI Hype Cycles

<p>Innovations in the field of Machine Learning (ML) continue to be driven by a dynamic research community. The ML team at Scale is proud to contribute to the field through its own research, dataset partnerships with leading Universities such as Oxford (<a href="https://scale.com/open-datasets/oxford">DEBAGREEMENT: Reddit 50K Dataset</a>) and the University of Waterloo and the University of Toronto (<a href="https://scale.com/open-datasets/oxford">CADC Dataset</a>), and collaborations with Stanford University, the University of California, Berkeley, and the University of Washington.</p>
<p>As <a href="https://neurips.cc/">NeurIPS 2021</a>, gets underway, we are pleased to share details on three papers the ML team have published as part of the conference:</p>
<ol>
<li>DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates
<strong>Authors:</strong> John Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, Doyne Farmer</li>
<li>Natural Adversarial Objects
<strong>Authors:</strong> Felix Lau, Sasha Harrison, Nishant Subramani, Aerin Kim, Elliot Branson, Rosanne Liu</li>
<li>Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning
<strong>Authors:</strong> Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, Ludwig Schmidt</li>
</ol>
<h2 id="debagreementacommentreplydatasetfordisagreementdetectioninonlinedebateshttpsopenreviewnetforumidudvun__gfo"><a href="https://openreview.net/forum?id=udVUN__gFO">DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</a></h2>
<h3 id="papersummary">Paper Summary:</h3>
<p>DEBAGREEMENT is a dataset including 42,894 comment-reply pairings from the popular debate website Reddit, each of which has been labeled with agree, neutral, or disagree. We gathered interactions from five different forums: r/BlackLivesMatter, r/Brexit, r/Climate, r/Democrats, and r/Republican. Comment pairings for each forum were chosen in such a way that they generate a user interaction graph when taken as a whole. </p>
<p>DEBAGREEMENT offers a challenge for Natural Language Processing (NLP) systems because it includes slang, irony, and topic-specific humor, all of which are common in online discussions. </p>
<p>We compared the performance of state-of-the-art language models on a (dis)agreement detection task, and looked at the usage of contextual information that is accessible to the models during training (graph, authorship, and temporal information).</p>
<p>DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect agreements as well as disagreements online, in light of recent research showing that context, such as social context or knowledge graph information, enables language models to perform better on downstream NLP tasks.</p>
<h3 id="keytakeaways">Key Takeaways:</h3>
<ol>
<li>Language models trained on existing datasets underperform when run on real world data. This discrepancy is exposed when evaluating the models using the new DEBAGREEMENT dataset.</li>
<li>DEBAGREEMENT presents new opportunities for modeling diverse online interactions with text and context (authorship, graph, temporal information). Its graph structure allows for the combination of text-based machine learning (ML) with graph representation learning (GRL) approaches.</li>
<li>By modeling online discussion forums as graphs of user interactions, researchers can: 1) transform the agreement/disagreement detection problem into a sign link prediction task, and 2) use existing signed graph embedding techniques evaluated on publicly accessible signed graphs such as Epinions and Slashdot.
The sentiment and polarization of topics is not static in time and for the first time, this dataset exposes that and shows how realistic topic sentiment evolves over time.</li>
</ol>
<h2 id="naturaladversarialobjectshttpsarxivorgabs211104204"><a href="https://arxiv.org/abs/2111.04204">Natural Adversarial Objects</a></h2>
<h3 id="papersummary-1">Paper Summary:</h3>
<p>Although cutting-edge object identification approaches have shown impressive performance, models are often vulnerable to adversarial assaults and out-of-distribution data.</p>
<p>Natural Adversarial Objects (NAO) is a novel dataset for assessing the robustness of object identification techniques. NAO comprises 7,934 images and 9,943 objects that are unaltered and depict real-world circumstances, yet cause cutting-edge detection methods to misclassify with high confidence. When compared to the conventional MS-COCO validation set, the mean average precision (mAP) of EfficientDet-D7 lowers by 74.5 percent when tested on NAO.</p>
<p>Furthermore, when evaluating a range of object detection architectures, we discovered that improved performance on the MS-COCO validation set does not always transfer to better performance on the NAO, implying that robustness cannot be gained merely by training a more accurate model.</p>
<p>We look at why NAO cases are tough to identify and classify. Experiments using image patch shuffles demonstrate that models are too sensitive to local texture. Furthermore, when predicting class labels with integrated gradients and background replacement, we discovered that the detection model is dependent on pixel information inside the bounding box and insensitive to background context.</p>
<h3 id="keytakeaways-1">Key Takeaways:</h3>
<ol>
<li>The NAO dataset presents a difficult robustness assessment challenge for detection models trained on MS-COCO. As part of our evaluation, we tested seven state-of-the-art detection models from a variety of families, and found that they consistently failed to perform accurately on NAO when compared to the MS-COCO validation set, which included both in-distribution and out-of-distribution objects, among other things.</li>
<li>We also described the process of building such a dataset yourself, which may be used to create similar datasets for any target dataset or pre-trained model.</li>
<li>The popular MS-COCO dataset has "blind-spots" that make it challenging to appropriately categorize these naturally adversarial objects. </li>
<li>Additionally, we show that detection models are overly sensitive to local texture but insensitive to background change, which makes them vulnerable to natural adversarial objects.</li>
</ol>
<h2 id="arewelearningyetametareviewofevaluationfailuresacrossmachinelearninghttpsopenreviewnetpdfidmpducs1msek"><a href="https://openreview.net/pdf?id=mPducS1MsEK">Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</a></h2>
<h3 id="papersummary-2">Paper Summary:</h3>
<p>Evaluation issues often undermine the validity of results in machine learning research. In collaboration with researchers from Stanford University, the University of California, Berkeley, and the University of Washington, we conducted a meta-review of 100+ survey papers to identify common benchmark evaluation problems across subfields. In some cases, several years’ worth of progress in certain fields may be misstated. </p>
<p>Our meta-review surveys evaluation papers reporting on a broad range of subfields, ranging from computer vision to deep reinforcement learning, to recommender systems and natural language processing, and more. We found a consistent set of failure modes, which we organized into a systematic taxonomy. This taxonomy of failure modes helps researchers understand where different types of failures are common and provides a framework for future research.</p>
<p>Our taxonomy distinguishes between “internal” and “external” validity failures. Internal validity failures occur within the context of a single benchmark and raise questions about the consistency or reliability of measurements on that specific benchmark. For example, running baseline methods with less hyperparameter tuning than a newly proposed method can underestimate the performance of existing methods and hence lead to overly optimistic claims of performance improvements. External validity failures, in contrast, describe failures that occur when dealing with multiple benchmarks. External validity failures occur when results from one benchmark do not generalize to other settings. One example is dataset misalignment: if data collected for a benchmark is not representative of the setting where models will be used, then results from that benchmark may not reliably predict performance in the real world. </p>
<h3 id="keytakeaways-2">Key Takeaways:</h3>
<ol>
<li>Evaluation failures pose a risk to the validity of research findings in machine learning. Our meta-review of 100+ survey papers observed common failure modes across a wide range of subfields. </li>
<li>While some internal validity failures can be identified in the context of a single benchmark, other external validity failures only appear when looking at multiple datasets.</li>
</ol>
<h2 id="conclusion">Conclusion</h2>
<p>As we look ahead to 2022, we look forward to supporting more university research teams and driving more research initiatives within Scale. To learn more about our work and the work of many of our partners, our team hosts regular TechTalks on <a href="https://exchange.scale.com/public/events">Scale Exchange</a>, our community of ML practitioners. Join the community to learn more and register for upcoming talks. If you’re interested in joining our growing ML team, take a look at our <a href="https://scale.com/careers">careers page</a> for open positions. </p> 

An Overview of ML Research Papers Accepted At NeurIPS 2021

Research Papers Accepted At NeurIPS 2021

Machine Learning Papers at NeurIPS 2021

<p>Artificial Intelligence (AI) is now a part of our everyday lives and has the potential to massively impact the future of our society. Leaders in the private sector are making a real-world impact with AI today - from autonomous vehicles, medical diagnostics, vaccine distribution, and eCommerce. We also recognize the importance of the federal government, academia, and the private sector to work collaboratively to drive innovation and ensure continued U.S. leadership in technology. </p>
<p>In early October, we hosted our second virtual conference: TransformX. Over the two days, we brought together more than 22,000 researchers, practitioners, and executives to hear from leaders across industries, academia, and government. We’re excited to share some key insights and takeaways that the foremost government leaders have on the future of AI.</p>
<h1 id="firesidechatswithkeyleaders">Fireside Chats With Key Leaders</h1>
<p>Mark “Drifter” Valentine, Head of Federal at Scale AI hosted fireside discussions with key government and private sector leaders including Lieutenant General Kirk S. Pierce, Commander, Continental U.S. North American Aerospace Defense Command Region-1st Air Force, Rear Admiral Lorin Selby, Chief of Naval Research, and Steven Escaravage, Senior Vice President at Booz Allen Hamilton.</p>
<h2 id="lieutenantgeneralkirkspierce">Lieutenant General Kirk S. Pierce</h2>
<p>LTG Pierce spoke with Mark on operationalizing AI for National Security. </p>
<p>For Pierce, homeland defense is a chief interest. To work effectively and with prudence, “We have this grand, sweeping concept of joint all domain command and control.” To achieve this, Pierce says interoperability is key. The key to interoperability? Data.</p>
<p>“We cannot continue to have data stovepipes,” he said. “Depending on wherever that data comes from—whether it's a sensor, above water, below water, in space, whatever it is, we want to have the data to be able to make decisions.”</p>
<p>He also said that he sees AI and ML operability as “an opportunity to reset, fundamentally, how we look at information, how we aggregate information and be able to evaluate and assess the validity of the information.”</p>
<p>“The companies that are working on that are gonna change how we fundamentally, as senior leaders, make decisions.”</p> <div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/vgxbm41ptn" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div><div class="markdown-module-blog"><h2 id="rearadmirallorinselby">Rear Admiral Lorin Selby</h2>
<p>RADM Selby discussed the importance of fostering an AI innovation ecosystem. </p>
<p>As a leader who is positioned at the cross-section of government, academia and industry, Selby is optimistic about the utility of AI to help humans reach their potential, and the Navy achieve its goals.</p>
<p>He foresees a future where AI can “unburden the mind” from tedious bureaucratic work, “So we can actually focus on the things the human mind is really good at focusing on— which is kind of more strategic thinking, more abstract thinking.”</p>
<p>“This is a pivotal moment in history,” he said of AI and ML advancements. “I think we’ll look back on this time as a key transition phase in history.”</p>
<p>Selby said that building confidence in AI systems is essential to innovation, as is remaining anchored in the strength of human decision making.</p>
<p>“We have to develop trust,” he said, while emphasizing the need to keep humans in the loop of critical decisions. </p></div><div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/h3hikuszel" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div><div class="markdown-module-blog"><h2 id="steveescaravage">Steve Escaravage</h2>
<p>Steve Escaravage shared his insights on developing realistic approaches to deploying machine learning (ML) in federal environments. </p>
<p>Escaravage said that for advancements in ML, the private sector is leading, and the government has to follow. <strong>“The innovation is coming from outside—it’s coming from the private sector, from commercial use cases”</strong> he said. “For the public sector—for the government to be successful, we simply cannot ignore that investment and we need to channel that into the public sector.”</p>
<p>He said that although there has been real progress and momentum since 2017, there are pieces of the puzzle that are missing. Friction points include: access to real world data assets and data sets to train AI, approved computing environments that are configurable and scalable and systems with authorization to operate (ATO) accreditation. 
“We all have a responsibility as citizens of our governments to try to get them to better outcomes,” he remarked.</p></div><div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/0b1pm4gn25" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div><div class="markdown-module-blog"><h2 id="expertpanel">Expert Panel:</h2>
<p>In addition to the fireside chats, Mark also led an expert panel discussion on The State of AI for National Security Mission Applications featuring Michael “Rabbi” Harasimowicz and Rachael Martin. </p>
<p>Martin said that she hopes that in the future, AI will be a seamless part of her agency’s workflow, and as commonplace, critical and functional as something like email. “The new warfighting demand is time,” she said. “When you have machine speed, you need to have trust and confidence that what the machine is telling you is correct.”</p>
<p>As the NGA relies on overhead imagery and data to compel decision-making, she said it’s important to have a systematic data workflow from input to warfighter. “We do think that AI and automation is really the best way of accomplishing that,” Martin said.</p>
<p>She added that she sees the time investment and the leadership investment in bringing AI to the mission space as heartening. “You have to change culture by getting people involved in the change,” she reflected.</p>
<p>Harasimowicz said that if our government doesn’t update its systems, the systems and the country are at risk. “Our national security challenges are really getting to the point where they're beyond human scale,” he said. “And I say that because the amount of data that is flowing in and around all of our domains is just insurmountable for just a commander.”</p>
<p>The level of support required to manage the data at a tactical and strategic level drives us to have a level of assistance, he said. “So artificial intelligence can provide that backdrop.”</p>
<p>“Our adversaries are challenging themselves in aggressive ways,” Harasimowicz said. “And we have to be as aggressive, if not more so aggressive…We have to go faster. We have to find ways to solve solutions in weeks, as opposed to years.”</p></div><div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/2yn8oyur22" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div>

Highlights from TransformX 2021: U.S. Government Related Sessions

Highlights from TransformX 2021: U.S. Government Sessions

<h1 id="naturallanguageprocessingandonlinediscourse">Natural Language Processing and Online Discourse:</h1>
<p>Online debates shape our society, and have a profound impact on how people perceive the world. Over 4.2 billion people actively use social media, and many of them participate in online discussions. The breadth and depth of online discourse provide insights and data for researchers and curious people alike.</p>
<p>One way to understand data from online discussions is through the field of Natural Language Processing (NLP). NLP offers techniques to understand textual interactions, and to gain a better understanding of polarization online.</p>
<p>However, current NLP models often struggle with nuanced language in online exchanges, such as slang, sarcasm and topic-specific jokes, making it difficult to understand diverse online interactions. A task called stance detection is useful in this arena to address this challenge.</p>
<p>Stance detection is a text understanding task that consists of classifying the position (or stance) of the author of a piece of text, towards an initial message or statement. In an online setting, these statements might include comments or messages in a thread. The stance is classified into one of three classes: in favor, against, or neutral.</p>
<h1 id="methodology">Methodology:</h1>
<p>Recent work has shown that stance detection would benefit from context-sensitive approaches. We were not able to find realistic, large, open datasets with contextual information which do not have <strong>platform-specific</strong> features. </p>
<p>Thus, we created <a href="https://scale.com/open-datasets/oxford">DEBAGREEMENT</a> with conversations from Reddit to better understand the evolution of online debates, and to provide annotated data for agreement and disagreement detection. </p>
<p>The comment or reply function on social media sites does not provide sufficient information to classify an interaction. So, we looked at actual online conversations. Unlike existing datasets for stance detection, DEBAGREEMENT provides realistic online discussions, with diverse writing styles, genres and topics of discussion.</p>
<p>The set is comprised of 42,894 comment-reply pairs from Reddit, annotated with <strong>agree, neutral, or disagree labels</strong>. </p>
<p>The dataset collects data from five forums on Reddit: </p>
<ol>
<li>r/BlackLivesMatter</li>
<li>r/Brexit</li>
<li>r/Climate</li>
<li>r/Democrats</li>
<li>r/Republican </li>
</ol>
<p>We pulled from three subreddits of social movements r/Brexit, r/BlackLivesMatter, r/Climate, and two subreddits of political affiliations r/Republican and r/Democrats.</p>
<p>To ensure we were only using high-quality interactions, we excluded: empty comments, comments from deleted authors, comments that were hidden for user privacy reasons, and comments containing hyperlinks. Further, in order to annotate impactful discussions in a given subreddit, we removed posts with fewer than 10 words and few comments.</p>
<p>This dataset presents opportunities to detect (dis)agreements by leveraging context beyond just text. The set does not rely on platform-specific features, such as retweets or hashtags. The data was collected such that, for each subreddit, the resulting set of interactions formed a multi-edge, temporal graph where nodes are users, and edges represent a comment-reply interaction between two users.</p>
<h1 id="challenges">Challenges:</h1>
<p>Despite the preprocessing we performed to ensure high-quality annotations, the dataset annotation came with certain obstacles. </p>
<p>We used methods that were developed in-house such as dynamic consensus to ensure high quality annotations. (For more details on dynamic consensus, take a look at our blog on <a href="https://scale.com/blog/ml-human-consensus">ML + Human Consensus</a>.) Annotating short online exchanges is tough; only 33% of the pairs were annotated with a full agreement among annotators. Accounting for nuanced statements, sarcasm, diverse writing genres, and specific socio-economic contexts is a tall order. </p>
<h1 id="results">Results:</h1>
<p>This research presents a challenge for NLP systems, and attempts to argue:</p>
<ul>
<li>Current language models struggle with messy data when it is either scraped directly from social media, or pulled from the human dialogue.</li>
<li>Clean and tailored datasets may not have transferable insights for online (dis)agreement detection.</li>
</ul>
<p>We evaluated the performance of state-of-the-art language models against our new stance detection dataset and assessed the transfer learning potential with other stance detection datasets already available. </p>
<p>State-of-the-art language models are faring well on simple examples, but fail on more complex ones. These models tend to overly rely on superficial cues, such as the first word of the reply. </p>
<p>Moreover, it appears that models trained on clean stance datasets behave poorly on our new dataset, which speaks to the new challenges that our dataset offers.</p>
<p>This research presents promising avenues. Our findings highlight ways to improve language models with contextual information.</p>
<p>We see promise in DEBAGREEMENT as it presents an opportunity to train socially aware language models.</p>
<h1 id="furtherresearchandrecommendations">Further Research and Recommendations:</h1>
<p>We believe expanding annotations to other areas of discussion, such as consumer products reviews, investor forums or wider political forums could lead to further discoveries and insights about future consumer preferences or investor decision-making. </p>
<p>The dataset also provides the possibility to investigate social theories, understand polarisation, study online social movements and examine how people express their opinions and change their views on social media.</p>
<h1 id="readthefullresearch">Read The Full Research:</h1>
<p>This work was achieved in collaboration with Oxford University, who are the first authors of the paper. </p>
<p>The paper has been accepted to the “Datasets and Benchmarks Track” of <a href="https://neurips.cc/">Neurips 2021</a>. The paper details each of these issues further and is accessible on <a href="https://openreview.net/forum?id=udVUN__gFO">OpenReview</a>.</p> 

Scale Text

Content & Language

New Research Paves The Way to Train Socially Aware Language Models

Scale AI And Oxford University's Reddit Dataset Provides Comprehensive Online Discourse Data

<p>At Scale, we help our customers accelerate the development of their AI applications. We do this through a variety of products and services, but here we’ll focus on one: high-quality labels for training data. We’ve developed a process for delivering model-improving data quickly and efficiently, and we highlight parts of this process below.</p>
<p>To move fast, we align with our customers early. We strive to understand what our customers ask for and why, so that we can use the best combination of product and operational expertise to label their data with high quality - with their ML goals in mind. We take a collaborative and thoughtful approach to architecting and building data annotation pipelines. It’s one of the many reasons that customers continue to trust Scale as their AI partner.</p>
<h1 id="wearchitectdataannotationpipelinesthatmaptoourcustomersmlgoals">We architect data annotation pipelines that map to our customers’ ML goals</h1>
<p>At Scale, we start by understanding what our customers need to accomplish. What specifically will the AI model need to do? Is it for an <a href="https://scale.com/customers/toyota">Autonomous Vehicle</a> (AV) use case, perhaps predicting the action of another agent? Or is it for an application that can recognize and track objects for <a href="https://scale.com/customers/cellareye">inventory management</a>? From this, we can begin to understand:</p>
<ol>
<li>What data will the model ‘learn’ from?</li>
<li>What data will be used for inference, once the model is deployed in production?</li>
<li>What should the ideal output look like?</li>
<li>What determines a good (or bad) inference output?</li>
</ol>
<p>We’ve found that the success of an AI project is highly correlated with the quality and specificity of the training data used. This means that a broad or general training dataset won’t usually be the most efficient or quickest way to build an ML model. As a result, we collaborate with customers to set up goal-oriented annotation pipelines that map directly to their ML roadmaps. This approach helps them achieve their KPIs and project milestones.</p>
<p>As an example, it is common for new customers to ask us to replicate a broad-purpose public dataset, such as the AV-centric <a href="https://scale.com/open-datasets/nuscenes">NuScenes</a>, which we released with <a href="https://www.aptiv.com/">Aptiv</a>. Once we deep dive on their ML roadmap however, we discover the sub-goals (such as lane planning and pedestrian intent recognition) and the desired timelines for each. We then build specialized pipelines that generate the required training data at higher quality and at faster speeds than the customer would otherwise have done themselves. In fact, our speed of operation has frequently helped customers meet key deadlines, like public conferences or demos.</p> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/673f8157-ef5b-4b51-a76d-ee5f8b9d0100/public" alt="Common Autonomous Vehicle ML Roadmap and Data Needs." />
        <figcaption>Scale seeks to understand our customers' timelines of ML goals and to spin up label pipelines targeted at the specific data needs for each goal.</figcaption></figure><div class="markdown-module-blog"><p>It’s important to note that few customers come to us with neatly-defined, single pipeline needs (they used to!). Today, our customers often bring multi-modal data from multiple sensors. For an AV customer, this could mean LiDAR, radar, and a combination of regular and fisheye cameras. Having a clear understanding of our customers' goals helps us be a thought partner on the types of objects or events to label across these sensors. It also helps us propose a pipeline architecture that incorporates the best product capabilities to produce a comprehensive, goal-oriented training dataset. </p>
<p>For example, we might propose a string of dependent pipelines that weave together categorization, 3D LiDAR cuboids, and LiDAR linking. Weather attributes and 3D data defects might be caught and categorized in the categorization pipeline, vehicles might be tracked in the 3D LiDAR cuboids project, and camera-based attributes like blinker status might be generated in the linking project.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e43fe86-4639-495c-3096-34676043a300/public" alt="Using specialized label pipelines to turn multi-sensor inputs into training data. " />
        <figcaption>AV customers often come to Scale with data from multiple sensors and with an idea of what labels they need. Scale's team then architects specialized label pipelines that bridge the two, producing high-quality labeled data quickly and efficiently.</figcaption></figure><div class="markdown-module-blog"><h1 id="wetestandfinetunepipelinearchitecturebycreatingagoldendatasetandvettingitwithourcustomers">We test and fine tune pipeline architecture by creating a golden dataset and vetting it with our customers</h1>
<p>Following this thoughtful approach to customer alignment helps us achieve a data milestone: the creation of a golden dataset. A golden dataset is used to define what ‘good’ looks like. It is used as a reference for us, our human Taskers, and customers’ own stakeholders.</p>
<p>Creating a golden dataset is a journey of discovery. We set out to create the golden dataset by annotating a representative sample of data in the pipeline we architected. Annotating some data always generates observations and insights about nuances in the data and requirements, which we surface to the customer along with our recommendations. We then review the resultant annotated data with our customer, tying open questions and considerations back to the customer’s ML goals.</p>
<p>Reviewing the proposed golden dataset together with the customer is key. For example, we were working with an autonomous drone customer who needed polygon labeling of flyable space, ie where their drones could safely navigate. We found ourselves drawing pixel-perfect polygons around every tree branch, only to have the customer reject every task. Only once we debriefed live with the customer did we realize that for them, quality training data meant polygons that avoided tree pixels by a wide margin - ie, rough, simple polygons that just didn’t clip the treetops! That one collaborative sync put the project back on track with a clear path to consistent, relevant, and accurate labeled data.</p>
<p>Attaining this mutually-vetted golden dataset is powerful. It powers our internal (human) Tasker training and assessment tools, giving both Taskers and the operations team a clear understanding of what success looks like. It also aligns our Quality Assurance (QA) team, whose job is to effectively mirror customers’ QA teams to reduce customers’ operational data-management burden.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ecd95dc4-c137-4a9e-4024-569dd8df3d00/public" alt="Achieving and leveraging golden data" />
        <figcaption>Reviewing label samples together with customers helps establish a golden dataset, which in turn makes is possible to scale up production of high-quality labels.</figcaption></figure><div class="markdown-module-blog"><h1 id="weengageallrelevantstakeholderstoensurecustomersarealignedwithscaleandamongstthemselves">We engage all relevant stakeholders to ensure customers are aligned with Scale - and amongst themselves</h1>
<p>We’ve recognized that our customers are running complex initiatives involving many stakeholders on many teams, including operations, engineering, legal, and procurement. Alignment isn’t complete until Scale has aligned with each customer subteam, AND the customer subteams are internally aligned amongst themselves. </p>
<p>Projects can fail if the customer’s operations team is grading labeled data according to a different set of standards than what the ML team expected. This can take many forms - operations might be grading too easily (for example, approving polygons that are too swaggy), or grading too harshly (for example, rejecting boxes with acceptable sizing), or just grading according to a different or outdated rubric (for example, applying an old definition of “truck” that included small pickup trucks). </p>
<p>The result of inconsistent labeling standards is that the ML team gets training data that doesn’t improve model performance. Or, the ML team gets starved for data, even though there is data available that would have improved performance! Few things are worse than delivering weeks or months of data, with a consistent green light from the customer’s operations team - only to get a mass rejection from the ML team once it tests model performance.</p></div><div class="markdown-module-blog"><p>Scale drives stakeholder alignment by bringing the right people into the discussion, especially at steps that we know to be high risk. We also actively monitor for hints that there might be misalignment (like radio silence from some stakeholders at key milestones) and work with our customers to drive alignment internally and with Scale. We pride ourselves at being a knowledgeable partner who knows how to avoid situations that de-ccelerate the development of AI applications.</p>
<h1 id="alignmentisneverwonanddonewecontinuetoreassessalignmenttoourcustomersmlgoals">Alignment is never won and done; we continue to re-assess alignment to our customers’ ML goals</h1>
<p>It takes a deliberate effort to maintain alignment. Very often, model evaluations lead customers to update their data requirements and labeling guidelines. Even when the labeling guidelines remain constant, customer data itself can change. For example, we once built a reliable high-quality pipeline to label street scenes, only to find the next batch of customer data contained scenes of people lying on a racetrack. </p>
<p>Throughout the lifecycle of the data annotation pipeline, we continue to apply our alignment process. We continue to check in on customer ML goals, refine pipeline setup, maintain the accuracy of the golden dataset, and ensure alignment between teams on the customer, Scale, and Tasker side. This accelerates the speed with which our customers receive high-quality labeled data, which in turn accelerates their development of AI applications.</p>
<p>To experiment with Scale’s data labelling at a small scale, sign up for <a href="https://scale.com/rapid">Scale Rapid</a>, the fastest way to production-quality labels, with no data minimums. To set up custom pipelines for your ML needs, <a href="https://scale.com/sales">talk to our team</a>. </p></div>

Scale Image

Autonomous Checkout

Robotics

Scale 3D Sensor Fusion

Manufacturing

Scale Video

Real Estate

Retail & eCommerce

Autonomous Drones

We strive to understand what our customers ask for and why,  to label their data at high quality.

Partnering with customers to architect data pipelines

How Does Scale AI Partner with Customers to Produce High-Quality Training Data?

<p>Every year, nearly 200,000 service members transition to civilian life. As a fighter pilot who served for over two decades in the U.S. Air Force, I know the challenges these men and women face. From adjusting to “normal” life, to mental health struggles, to determining a new career path, it can be overwhelming and difficult to navigate. </p>
<p>One of my favorite things about Scale is our CEO’s hiring philosophy. Simply put, he says to hire people who give a shit. That philosophy has brought an amazing group of people to this company who not only care about technology, but also care about our country, our government, and our service members.</p>
<p>As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories to honor them and their commitment to our country, and to also offer advice for those navigating the transition from service to civilian life. </p>
<h1 id="whatdidyoudointhemilitary">What did you do in the military?</h1>
<p><strong>Jason Schmid</strong>: I was a U.S. Army Military Intelligence Analyst, and deployed to Iraq twice. The first time was as a part of the Iraq Survey Group, determining the disposition of Iraq’s suspected weapons of mass destruction programs. On that mission, I was part of the team interrogating former high ranking Iraqi Regime officials. On my second deployment, I was embedded as a Tactical Intelligence Trainer in an Iraqi infantry battalion. </p>
<p><strong>Dan Tadross</strong>: I was an Air Traffic Control Officer responsible for coordinating aircraft requests and the deconfliction of airspace. In my last few years of Active Duty, I had the opportunity to serve as an Innovation Advisor to the Secretary of the Navy on how to integrate AI into Aviation Command and Control (specifically how to shorten planning cycles). I also had the privilege of being a founding member of the Department of Defense (DOD) Joint Artificial Intelligence Center (JAIC). During my two years at the JAIC, I ran a number of programs from supporting testing and evaluation to establishing Project Gargoyle for Force Protection.</p>
<p><strong>Brianna Hester</strong>: I was a Geospatial Analyst for 24 years. My most recent position was the Operations Superintendent of the European Partner Integration Enterprise (EPIE) where I managed a 40-person, international intelligence team operating a $5.7M Intelligence, Surveillance, Reconnaissance (ISR) coalition program for combat mission planning, execution, and overall intelligence gathering and research. I oversaw ISR mission operations and imagery exploitation of U.S. ISR platforms employed for collection of full motion video imagery in support of the DOD, NATO, and coalition forces. I deployed to Qatar and Afghanistan and also traveled to numerous other countries for my duties. </p>
<p><strong>Jacob Sheehan</strong>: I was a U.S. Army Special Forces Officer, also known as a Green Beret, where I deployed on combat missions to Iraq and Afghanistan and on training missions in India, Thailand, and Korea. In addition to leading operational teams, I also had an academic assignment at West Point where I taught economics courses, ran the graduate scholarship program (Rhodes, Marshall, Fulbright, etc.) and designed a two-semester critical thought curriculum. </p>
<p><strong>Matt Beck</strong>: I was commissioned as a Naval Officer in 2008. I spent my first two years as a Surface Warfare Officer on a destroyer. I was selected to attend SEAL training after that tour and completed training in 2012. I led teams of various sizes on deployments to Europe, Africa, South America, and the Middle East.</p>
<p><strong>Tyler Haley</strong>: I was a Special Operations Communicator in the U.S. Army for 10 years. I spent most of my career inside of Special Operations doing everything from tactical communications to designing and deploying data centers. I’ve spent a large portion of my career in support of overseas operations from Afghanistan, Iraq, Syria, to most of Central Africa. I was lucky enough to get to finish my career in doing what I love most, which is any and all things cloud!</p>
<h1 id="whyscale">Why Scale?</h1>
<p><strong>Jason Schmid</strong>: When I left government service, it was important to me to join a company that values its contribution to national security. All of the Scale senior leaders I spoke to in the interview process made it clear that Scale is dedicated to using its technology to serve the nation. I think it is hard to overstate how important the work Scale is doing for the nation, both on the commercial and federal sides of the business. </p>
<p><strong>Dan Tadross</strong>: One of the most challenging aspects for the U.S. government to adopt and implement AI responsibly is the lack of structured data. Scale is at the forefront of developing structured data and can do so at a volume and quality that can help catapult government efforts and ensure the competitive edge for the U.S. </p>
<p><strong>Brianna Hester</strong>: I was intrigued by the job description. When I opened it and looked at the requirements, I had all the requested experience so I applied. After chatting with Austin Woo about some more specifics of the position, I knew I wanted to be a part of this program. </p>
<p><strong>Jacob Sheehan</strong>: I was primarily drawn to Scale through my hiring manager, David Kinsella. I was headed towards working at a financial technology company in my job search, but David introduced me to Scale’s compelling vision in the AI industry and the role it could play in making the U.S. government more nimble to respond to pressing challenges. He viewed my military experience as an asset and imagined my future impact at Scale rather than solely focusing on my non-traditional business operations background.  </p>
<p><strong>Matt Beck</strong>: I was drawn to Scale’s mission of accelerating the adoption of AI. A lot of the work we are doing on the federal team has a direct application to our nation’s defense, and I wanted to be a part of that work. </p>
<p><strong>Tyler Haley</strong>: I love the team dynamics of start-ups. Everyone wears different hats depending on the meeting or the time of day. This is super relatable to being on small unit teams inside of the Special Operations community. Scale, in particular, pushes the boundaries inside of technology and solves hard problems, which is exactly what I did in the U.S. military.</p>
<h1 id="whatdoyoudoatscale">What do you do at Scale?</h1>
<p><strong>Jason Schmid</strong>: I am Scale’s Head of Government Relations. I work to help Congress understand AI and ML capabilities so that they can build programs and policies that will help the U.S. government accelerate the adoption of AI. </p>
<p><strong>Dan Tadross</strong>: I am an Engagement Manager helping to bridge the gap between U.S. government requirements and experimentation and Scale’s expertise. </p>
<p><strong>Brianna Hester</strong>: I am the Synthetic Aperture Radar (SAR) Imagery Intelligence Subject Matter Expert (SME).</p>
<p><strong>Jacob Sheehan</strong>: I’m a Business Operations and Strategy Manager that supports the federal team. My main projects include growing a compounding network of labeling experts and labeler performance management.  </p>
<p><strong>Matt Beck</strong>: I am a Federal Deployment Strategist focused on developing solutions to support the modernization of government systems.</p>
<p><strong>Tyler Haley</strong>: I am the Infrastructure and Platform Lead for our federal team. I partner closely with our software engineering and security teams to help design infrastructure that works on various AWS classification regions.</p>
<h1 id="howwasyourtransitionfromservicetocivilianlife">How was your transition from service to civilian life?</h1>
<p><strong>Jason Schmid</strong>: I transitioned from the Army 15 years ago, and the circumstances of my transition were pretty unique. I went from being in literal combat to being a civilian in about three weeks. I was very fortunate to have a job lined up already, but it would have been much better if I had time to take advantage of all of the resources available to transitioning service members. Such a jarring transition was not easy. We now know so much more about successfully transitioning veterans back into civilian life that would have made my transition so much smoother - not just for me, but for my family as well. </p>
<p><strong>Dan Tadross</strong>: I transitioned off Active Duty service after 12 years during the height of the pandemic in 2020. So the transition process was stressful and challenging, but when I eventually landed at Scale my level of apprehension subsided.</p>
<p><strong>Brianna Hester</strong>: My transition was pretty easy, as I was ready to begin my post-military life. The hardest thing to get used to was not having to tell someone my every move. Taking PTO without telling someone the address of where I would be for the duration of the PTO was a little unnerving the first time.  </p>
<p><strong>Jacob Sheehan</strong>: I gave myself sufficient time, but there seems to be no such thing as too much time when it comes to major life transitions, especially if you have a family to take care of! I’m glad I had the space to think about what I wanted to do next instead of having to manufacture it quickly. Transitioning during COVID was a challenge from the standpoint of young kids being around all the time, so I would recommend finding a quiet space to do your transition thinking.</p>
<p><strong>Matt Beck</strong>: I had a wonderful transition departing the military. I met Shands Pickett early on in the process and kept in touch as positions at Scale became available. My preparation and exploration of many different post-military careers went well and resulted in me working at Scale. </p>
<p><strong>Tyler Haley</strong>: Transitioning was a bit tough for me. I got out of the service in May 2020 right as the pandemic was really setting in. Transitioning from being in a room with people that you are close with to only seeing people via Zoom and Slack was difficult at first. I was accustomed to seeing my daily contributions make a real impact, but when I got out, that faded away rather quickly. The good part about transitioning was being able to be at home and start a family. My wife and I welcomed our first kiddo into the world shortly after I got out. It gave me a new purpose and perspective in life. It helped me fill the void that the military left.</p>
<h1 id="canyouspeaktoyourexperienceasaveteranatscale">Can you speak to your experience as a veteran at Scale?</h1>
<p><strong>Jason Schmid</strong>: I think one of the best qualities of Scale’s culture is a genuine curiosity and passion to solve hard problems as a team. I think anyone who has served can connect with that. </p>
<p><strong>Dan Tadross</strong>: One of the aspects I miss most about serving in the U.S. Marine Corps is the comradery and sense of purpose. While nothing can compare with serving in a Marine unit, Scale has a culture that shares some similarities and the team I am a part of is motivated to succeed and support the U.S. government toward the responsible adoption of AI. </p>
<p><strong>Brianna Hester</strong>: Scale has become my new family. I still have my military family and always will but now I have added another branch to my family tree with everyone from Scale. They may be very different families but I wouldn’t trade either of them for anything and I can’t wait to see how much we accomplish together!</p>
<p><strong>Jacob Sheehan</strong>: Spending time with the many veterans on the federal team has maintained some of the camaraderie I enjoyed in the U.S. Army. I’m also pleasantly surprised how genuinely curious Scaliens across all of Scale are in veteran experiences! </p>
<p><strong>Matt Beck</strong>: Scale is extremely supportive of its veterans. My work keeps me engaged on a mission I believe in. Furthermore, Scale is extremely flexible in letting me maintain my connection to the U.S. Navy by supporting my drill requirements.</p>
<p><strong>Tyler Haley</strong>: Although I am still new to the team at Scale, I felt value-added from day one. As I mentioned earlier, feeling like what you do matters is a huge point of pride for most veterans. I instantly felt as if I was a part of the team and also was surrounded by other team members that have served. At Scale, you really get the experience of being back on those small teams that have a mission and purpose. </p>
<h1 id="whatadvicewouldyousharewithveteransmakingthetransition">What advice would you share with veterans making the transition?</h1>
<p><strong>Jason Schmid</strong>: Take advantage of every resource available to support your transition, whether you think it applies to you or not, because you’ve earned it. It’s normal for recently separated veterans to feel some loss of purpose. It’s important to find ways to continue serving something larger than yourself, even if it's not part of a career path. </p>
<p><strong>Dan Tadross</strong>: There are a lot of opportunities for those who want to find alternative ways to continue to support the U.S. government. Many of the skills I learned in the U.S. Marines continue to help me to succeed in the private sector. Those experiences and capabilities that made you successful as a service member are in demand in the private sector, so when making the transition, don’t forget the value you bring to any organization. </p>
<p><strong>Brianna Hester</strong>: My recommendation would be to take a break and relax for a minute before jumping right into another job, if you can. It’s a huge change going from complete structure in your everyday life to no structure, and it can be hard on some people. Enjoy a small (or even large) break and use that time as a way to prepare yourself for the great things to come. </p>
<p><strong>Jacob Sheehan</strong>: Spend a lot of time exploring the industry you want to work in, understanding the companies uniquely positioned in that industry, and thinking about the strengths you can bring to roles in those companies.  I’ve had many veteran friends get it backwards by over-indexing on the role (title) and barely giving any thought to the company or larger industry. </p>
<p><strong>Matt Beck</strong>: Always seek to improve your position as you look for opportunities. Forecast your worst case scenario out as far as possible. Once you are comfortable with your worst case scenario, seek out the roles and companies that you are most excited about and get connected early.</p>
<p><strong>Tyler Haley</strong>: Figure out what fills your voids. Find a routine and stick to it. A lot of the service members that I speak with feel the same way. Whether that’s getting up every morning and hitting the Peloton or cooking dinner for the family every night, find something that you can do for YOU. Find a new skill or hobby and try to master it. Get out of the house and away from the keyboard. Find time in your days to step away and get lunch with another veteran. Transitioning is different for everyone. Make time to see those who understand what you are going through. </p>
<p>If you are interested in joining our team, check out our <a href="https://scale.com/careers#jobs">open positions</a>. </p> 

As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories.

Scale Team Spotlight: Veterans at Scale

<h1 id="whybuildanmlbasedworkflowfordatalabeling">
  Why build an ML-based workflow for data labeling?
</h1>
<p>
  Delivering high-quality data labels quickly is a challenging task. Even for
  humans, image and video data commonly found in self-driving or robotic
  applications can be compositionally challenging to parse and correctly
  annotate. At Scale, we use machine learning to augment human workflows to both
  improve the quality and speed of labeling. Since deep learning models can
  struggle to perform consistently across diverse data domains like self-driving
  car scenes, finding the optimal balance between ML automation and human
  supervision is necessary to ensure we maintain high-quality.
</p>
<p>
  <em>Autosegment</em>, an ML-powered tool that can accurately segment instances
  within a human-defined bounding box, allows us to delegate high-level
  reasoning about semantics and taxonomy to Taskers (human labelers) and basic
  pixel-level processing to an ML model. This division of labor enables us to
  build a general purpose and accurate ML labeling tool. Despite using a simple
  neural network, <em>Autosegment</em> outperforms prior tools.
</p>
<h1 id="thechallengeofsemanticsegmentationlabeling">
  The Challenge of Semantic Segmentation Labeling
</h1>
<p>
  Semantic segmentation involves human Taskers identifying and labeling every
  pixel in a visual scene — a notoriously time-consuming and tedious process.
  Most Tasker time in this manual approach — about 95% — is consumed by
  time-intensive work, such as carefully tracing the outlines of objects in a
  scene. For example: it takes roughly one minute to trace the outline of a car
  and only three seconds to click on that outline and select the label “Car.”
  What if we could instead use machine learning to automate the most laborious
  parts of semantic segmentation? What if an ML model did the semantic
  segmentation, and the Tasker only had to select the right label?
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/72c723b3-46ae-4384-b75b-4a87b5e40b00/public"
    alt="Semantic Segmentation Task"
  />
  <figcaption>
    A visual scene which has been semantically segmented. Each pixel is color
    coded to denote the class of object it represents, i.e. Pedestrian, Road,
    Bike, etc.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    As ML engineers, we rely heavily on quantitative validation metrics, such as
    label accuracy and intersection-over-union (IoU), to determine the quality
    of our models. However, when building models that optimize for a particular
    business outcome — for instance, reducing the time it takes Taskers to
    semantically segment images — we need to think critically about which model
    to use and how we measure its performance..
  </p>
  <p>
    In this example, our objective is to produce predictions that streamline the
    Taskers’ job. This is not necessarily the same as “best model performance.”
    For example, an instance segmentation model might be consistently inaccurate
    by just a few pixels around an object. This model might have a high IoU
    score but very little practical usefulness because Taskers would need to
    re-do every trace of the outline.
  </p>
  <p>
    Conversely, a model that can perfectly segment an object, even if it only
    identifies half of it, would still be extremely useful. Despite having lower
    IoU validation scores, it’s still an improvement to the overall workflow.
    Because it takes so long to trace outlines manually, having a model produce
    pixel-accurate partial outlines is still a significant time saver.
  </p>
  <h1 id="theactivetoolingparadigmandtheautosegmenttool">
    The Active Tooling Paradigm and the Autosegment Tool
  </h1>
  <p>
    <em>Autosegment</em> is a machine learning tool that segments the most
    salient object within a human-provided box annotation. It incorporates
    Tasker feedback at annotation time in a way that is flexible and iterative.
  </p>
  <p>
    When building <em>Autosegment</em>, we realized early on that the only way
    to provide flexibility and high quality was to give Taskers access to Active
    Tools.
  </p>
  <p>
    Active Tools let Taskers make changes to the model inputs and parameters in
    real time to generate the best output for their use case. This is similar to
    “prompt engineering,” the process by which we can coerce a neural language
    model like GPT-3 to generate desired text when given precise prompts. For
    more information, OpenAI hosts
    <a href="https://beta.openai.com/docs/guides/completion/prompt-design"
      >excellent documentation</a
    >
    on how to design prompts for their GPT-3 model.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/07588dc3-6a06-45e7-c61f-a51562470900/public"
    alt="Open-AI GPT-3 Model"
  />
  <figcaption>
    Humans working alongside models can produce better results faster than
    models alone
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    Using Active Tools, we give Taskers as many options as possible to make
    their tasks easier. For example, they can adjust the annotation box shape
    and the prediction confidence threshold to configure the model to speed up
    their particular labeling task.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://cdn.builder.io/o/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ffef154571d5846b6b87674a46a123ac5%2Fcompressed?apiKey=e0438815ba51486bbb6a202747122d4b&token=fef154571d5846b6b87674a46a123ac5&alt=media&optimized=true"
    alt="Autosegment Tool Demo"
  />
  <figcaption>A demo of the Autosegment tool.</figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    Enabling humans to freely explore the output space of the model is a key
    advantage of active tooling. We've already seen how this is useful.
  </p>
  <p>
    Overall, we have observed the median annotation time per scene is about 30%
    faster with <em>Autosegment</em>. We also interviewed Taskers to see how
    they are using the tool. While our model is trained to segment cars and
    motorcycles, one Tasker used the tool to segment everything. While this
    wasn’t a workflow we had envisioned, it halved their time per scene
    segmentation, from 40 minutes to 20 minutes. In a discussion we had with
    five Taskers, they explained that they consider Autosegment to be more
    accurate and efficient than previous ML solutions. Now we can simply supply
    robust and general online models and let Taskers optimize the rest.
  </p>
  <h1 id="theautosegmentmodel">The <em>Autosegment</em> Model</h1>
  <p>
    <em>Autosegment</em> is a convolutional neural network trained to segment
    the most salient instance within a Tasker-defined bounding box. This simple
    design allows us to significantly reduce our customers’ operational overhead
    to deploy ML, while providing pixel-perfect instance segmentations, even on
    previously unseen instance types.
  </p>
  <p>
    The <em>Autosegment</em> model does not have to be particularly complex to
    produce compelling results. Autosegment has no notion of different semantic
    types; instead, it abstractly identifies and segments based on scene
    composition.
  </p>
  <p>
    The model is trained to always generate a segmentation. The training data
    includes a few key classes like pedestrians and vehicles. The task is
    binary; it simply determines whether each pixel belongs to the salient
    object or not. We don’t train the model with negative samples. Instead, the
    model makes its best guess on each pixel to generate a segmentation.
  </p>
  <p>
    We define the salient instance as the object that owns the central pixel of
    the crop, so even if the scene is cluttered, the model can identify the
    target instance to segment. This binary segmentation task across instance
    classes and types allows the model to generalize well on new classes and
    taxonomies. Indeed, we can transfer not only between objects of the same
    class, such as cars in different countries, but between different classes
    altogether — for instance, moving from cars to motorcycles to street signs.
    This transfer can be achieved with limited retraining or, in some cases, no
    retraining at all, achieving zero-shot transfer.
  </p>
</div>
<div class="player-wrapper-blog">
  <iframe
    src="https://scaleai.wistia.com/embed/medias/1a4x5jij9v"
    frameborder="0"
    allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
    allowfullscreen
  ></iframe>
</div>
<div class="markdown-module-blog">
  <p>
    Using salient object detection on specific crops of the scene can generate
    even more accurate segmentation. We were able to double the edge precision
    score from 0.26 to 0.52 when compared to off-the-shelf full-scene semantic
    segmentation models.
  </p>
  <p>
    Instead of supplying a downsampled version of the whole image to a neural
    network, the selected region is fed to the model at full resolution. Because
    the input is smaller, fewer pooling operations are required to achieve a
    globally receptive field. As a result, there is less downsampling and
    blurring. Our model achieves better fidelity around the edges of instances,
    a crucial part of providing high-quality data annotations to our customers.
    Improving full-scene labeling to the same degree would require exponentially
    more effort and complexity.
  </p>
  <p>
    In summary, we developed Active Tooling to create an ML-based workflow
    capable of solving basic computer vision problems. This workflow helps our
    Taskers reach higher levels of quality faster than in previous workflows.
  </p>
  <p>
    Innovations like <em>Autosegment</em> are critical to enable the “Moore's
    Law of data labeling.” At Scale AI, we’re excited to push the boundaries of
    human-AI interaction for our customers.
  </p>
</div>


Scale AI accelerates image annotation with semantic segmentation active tooling. 

Reducing image labeling time by up to 30% and doubling segmentation quality.

Faster Semantic Segmentation with Machine Learning-Assisted Workflows

<p>AI is no longer the exclusive domain of large enterprises. It’s entering the business mainstream—but it’s successful only sometimes, says Salesforce President and Chief Operating Officer Bret Taylor. A thoughtful AI strategy should be customer-focused, with clearly defined business outcomes, and the key to all of this is data.</p>
<p>In his talk at the <a href="https://scale.com/events/transformx"><a href="https://scale.com/events/transformx" target="_blank" rel="noopener noreferrer">Scale TransformX 2021 conference</a></a>, Taylor discussed the  key considerations that businesses of all sizes should keep in mind to ensure that they delight their customers and achieve sustainable business outcomes.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0;"><iframe src="https://fast.wistia.com/embed/medias/42dyrzqs4c" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe></div>
<p>Here’s a summary of the seven key takeaways from Taylor’s presentation. For more details, watch the session video above.</p>
<h1 id="businesstodayisallaboutthedata">Business Today Is All About the Data</h1>
<p>“If you don't have a digital business, you don't have a business,” says Taylor.</p>
<p>Enterprises need to maintain constant engagement with customers, supply chains, partners, and employees, in granular detail. Each of these interactions produces data that businesses can use to create more intelligent and personalized relationships. This <a href="https://insidebigdata.com/2017/02/16/the-exponential-growth-of-data/">growing tsunami of data</a> requires intelligent algorithms and automation to make sense of it all. As Marc Andreessen, co-founder of venture capital firm Andreessen Horowitz, famously said, <a href="https://a16z.com/2011/08/20/why-software-is-eating-the-world/">“Software is eating the world.”</a></p>
<p>Today, however, it’s AI that has become central to everything that a company such as Salesforce does.</p>
<h1 id="buildadataculture">Build a Data Culture</h1>
<p>AI and deep learning can’t be the exclusive domain of technology companies. To get there, every company must strive to <a href="https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Analytics/Our%20Insights/Why%20data%20culture%20matters/Why-data-culture-matters.ashx">develop a data culture</a> that allows employees to see and understand the flow of information moving through the business. A <a href="https://www.gartner.com/smarterwithgartner/do-you-need-a-chief-data-officer">chief data officer (CDO)</a> should act as a steward for all that information, without trapping data in internal silos. Instead, everyone needs to be empowered to make data-informed decisions based on objective views of customers, supply chains, and the flow of resources.</p>
<p>Ultimately, companies that create open data cultures will easily outcompete those that don’t.</p>
<h1 id="ethicalaibuildstrust">Ethical AI Builds Trust</h1>
<p>Shifting toward a data culture requires building in <a href="https://www.salesforce.com/blog/what-is-ethics-by-design/">ethics by design</a>, in much the same way that enterprises have incorporated <a href="https://patchstack.com/security-design-principles-owasp/">security by design</a>.</p>
<p>To deliver an ethical deployment of AI, the number one value must be trust. With that in mind, Salesforce created the position of <a href="https://www.cnbc.com/2018/12/10/salesforce-hires-its-first-chief-ethical-and-humane-use-officer.html">chief ethical and humane use officer</a> to guide the company’s adoption of AI.</p>
<p>Salesforce wanted to ensure that all of its AI is responsible, accountable, transparent, empowering, and inclusive. Taylor contends that failing to take this into account from the start inevitably leads to unintended negative consequences and general degradation of trust in technology.</p>
<p>Society is now holding software and AI to a higher standard. Companies that develop AI today must embrace conversations about ethics and trust, and they must embed those conversations into their development processes. Building trust also requires bringing in stakeholders who aren’t data scientists to help shape the responsible use of these technologies.</p>
<h1 id="focusonstrategynotaiforaissake">Focus on Strategy, Not AI for AI’s Sake</h1>
<p>While every company needs to become an AI company to remain competitive, too many organizations are succumbing to the hype about digital transformation without having <a href="https://www.thetilt.com/content/stages-of-digital-transformation">clear goals</a> in mind. According to the <a href="https://www.gartner.com/en/information-technology/insights/cio-agenda">2021 Gartner CIO Agenda survey</a>, while 94% of CIOs are working to deploy AI, only a quarter have done so successfully.</p>
<p>“Don’t start with the technology; start with customers,” Taylor says. Focus on their experience and what your customers need and want. Clarify the project’s scope and look for the easy wins first. “If AI becomes the purpose of the project, it becomes a bottomless pit of investment,” he says.</p>
<h1 id="oneshortcuttogoodaidontbuilditbuyit">One Shortcut to Good AI: Don’t Build It; Buy It</h1>
<p>Salesforce’s <a href="https://www.salesforce.com/products/einstein/faq/">AI platform Einstein</a> is a prime example of Salesforce focusing on building AI technology that is customer-focused and that “can be turned on with clicks, not code,” says Taylor. Salesforce customer <a href="https://www.salesforce.com/blog/orvis-ai-customer-engagement/">Orvis uses Einstein</a> to do simple analyses of the potential engagement of its marketing messages prior to their publication.</p>
<p>That approach increased Orvis’ click-through rate by 22% and boosted traffic by 30%. This is one way an enterprise can deploy customer-focused experiences with the help of AI, without having to invest in the substantial skills and technology normally associated with developing it.</p>
<h1 id="alignyouraiinvestmentwithbusinesspriorities">Align Your AI Investment with Business Priorities</h1>
<p>A common key strategy, especially for B2C enterprises, starts with identifying use cases that increase personalization and automation, thus streamlining the customer experience. Then, identify the data and AI tools that could help create those use cases.</p>
<p>For its part, Orvis’ goal is to keep its support calls short, and <a href="https://www.salesforce.com/news/press-releases/2019/11/19/hey-einstein-salesforce-brings-voice-to-every-customer-experience/">Einstein can help by transcribing and analyzing customer questions</a> and suggesting answers to call center personnel in real time.</p>
<p>The lesson here: Bring in just a few machine learning experts and combine them with off-the-shelf SaaS platforms to discover unique value in AI that can be customized around your business.</p>
<h1 id="dontforgetthequickwins">Don’t Forget the Quick Wins</h1>
<p>Getting value from AI isn’t science fiction, Taylor says. Simply step back and focus on the low-hanging fruit to achieve tangible business outcomes.</p>
<h2 id="learnmore">Learn More</h2>
<p>To learn more about how Salesforce built its AI strategy and lessons learned, <a href="https://exchange.scale.com/public/videos/enterprise-ai-strategy-bret-taylor-on-lessons-learned-at-salesforce">watch Taylor’s talk and view the full transcript here</a>.</p> 

A thoughtful AI strategy should be customer-focused, with clearly defined business outcomes, and the key to all of this is data. 

Effective AI Depends on Good Data, With Clearly Defined Business Outcomes Focused on Customers - Transformx 2021 Conference - Scale AI

7 Elements of a Well-Thought-Out AI Strategy: Salesforce COO Bret Taylor

<p>Today we’re thrilled to announce a new partnership with <a href="https://blend.com/">Blend</a>, a leading fintech company whose cloud banking platform is re-architecting banking software around the consumer. Through this partnership, we’re bringing the most advanced machine learning capabilities to Blend’s mortgage origination process, enabling easier and faster lending decisions for home buyers. </p>
<p>Traditional mortgage lending processes take 30 days to complete on average, and in today’s world where consumers can access goods and services instantly, they are demanding the same kind of efficiency from their financial institutions. </p>
<p>Scale’s Document AI platform will provide Blend with state of the art, fine-tuned machine learning models to process and extract information from complex mortgage documents in seconds. Our models do not require templates or rules, increasing the operational efficiency for our customers. And our fast human-in-the-loop review process ensures we’re maintaining 99%+ accuracy to reduce processing times to less than two days.</p>
<p>“Blend focuses on delivering a delightful, accessible and intuitive lending experience to consumers and robust, scalable, dynamic tools for bankers.” said Jeff Braddock, Business Development at Blend. “The combination of the Blend platform with Scale’s DocumentAI ensures the swift, accurate extraction and validation of data from documents, enabling bankers to make data-driven decisions with confidence.”</p>
<p>“In the past, incumbent financial institutions have not had great AI and ML technologies at their disposal that are simple to implement and easily fit within their existing technology stacks, so they’ve been stuck trying to build these capabilities in-house,” said Melisa Tokmak, GM of Document AI at Scale. “Blend is taking a truly customer-first approach by building best-in-class financial solutions and partnering with Scale to bring deep ML expertise that would take years and significant resources to develop alone. Ultimately this brings better, smarter solutions to their customers faster.”</p>
<p>Blend represents the growing use cases for Scale’s Document AI, which has quickly grown to serve customers across wide-ranging industries including financial services, logistics, and healthcare. To learn more about Document AI and schedule a demo, please view the <a href="https://scale.com/document-ai">Document AI webpage</a>. </p> 

Document Processing

Financial Services

Scale and Blend deliver faster lending decisions thanks to Scale Document AI.

Customer-centric lending platform Blend partners with ML expert Scale AI to make smart and instant lending decisions.

Blend and Scale AI: Leveraging ML to Make Smart, Instant Lending Decisions

Dr. Li shares explores how recent AI research helps robots  perform both short-horizon and long-horizon tasks.

Dr. Fei-Fei Li On Teaching Robots Complex Tasks - TransformX 2021 Conference - Scale AI

On Teaching Robots To Perform Complex Tasks with Dr. Fei-Fei Li 

<p>Today we are excited to welcome SiaSearch, a leading data management platform, to Scale. We acquired SiaSearch, a spin-off of the European AI venture studio Merantix, to enable us to expand Scale’s suite of tools for every AI team, broaden our footprint in European markets, and deepen the technical expertise of the company. </p>
<p>We know that the autonomous vehicle (AV) industry is one of the best breeding grounds for AI technology. SiaSearch’s Berlin-based team brings experienced, high-caliber engineers from the European AV industry to Scale, which will help us deliver world-class AI solutions to support the complete machine learning lifecycle. The SiaSearch team will join Scale's Nucleus business unit as we grow our platform to support more data types and use cases across a wide array of industries and markets.</p>
<p>SiaSearch’s unique interval querying engine, which is inspired by algorithms originally designed for large-scale analysis of DNA sequences, allows for lightning-fast search across massive video and LiDAR datasets and complements what Scale is building with <a href="scale.com/nucleus">Nucleus</a>. We are excited to weave the SiaSearch technology into the Nucleus product, accelerating development and improving the developer user experience. We see even more potential down the line to incorporate SiaSearch’s world-class team and technology throughout our entire suite of products. </p>
<p>Merantix and the SiaSearch team have a strong network in Europe and have partnered with the world’s leading automakers, including Porsche and Volkswagen. We’re thrilled to expand Scale’s footprint to Germany and look forward to working with SiaSearch’s customers and networks to significantly increase our ability to serve European-based customers. </p>
<p>We’re excited for what the future holds with this latest addition to our team, and we’re always looking for new opportunities. If you are interested in joining this incredible team, check out our <a href="https://scale.com/careers#jobs">open positions</a>. Or, if you’re interested in partnering with us, please <a href="https://scale.com/sales">reach out</a>.</p> 

Scale has acquired SiaSearch to further enhance Nucleus, its data management platform.

The SiaSearch team and Nucleus will collaborate on Dataset Managment for Nucleus.

SiaSearch Acquisition: Delivering the Full Data Engine to Our Customers

Waymo’s co-CEO Dmitri Dolgov shares Waymo’s approach to building powerful machine learning models for self-driving vehicles

Dmitri Dolgov of Waymo Discusses Data-centric AI - TransformX 2021 Conference - Scale AI

Dmitri Dolgov of Waymo Discusses Data-centric AI

The co-founder of Schmidt Futures and a former CEO of Google, discusses how AI will shape our global future. Eric was joined by Scale AI CEO Alexandr Wang

Eric Schmidt Discusses the Geopolitics of AI - TransformX 2021 Conference - Scale AI

Eric Schmidt Discusses the Geopolitics of AI

Justin Basilico, Director of Machine Learning and Recommender Systems at Netflix describes how ‘everything at Netflix is a recommendation’.

Netflix on Recommendations and Personalization - TransformX 2021 Conference - Scale AI

Netflix Explains Recommendations and Personalization

Clement Delangue HuggingFace CEO, shares what Transformers can do. He explores how we might ensure they are used ethically and transparently.

Clement Delangue HuggingFace CEO on Transformers - TransformX 2021 Conference - Scale AI

HuggingFace CEO, Clement Delangue: Are Transformers Becoming the Most Impactful Tech of the Decade?

<p>Today we’re thrilled to share that Nucleus now supports 3D Sensor Fusion data, including LiDAR point clouds. </p>
<p><a href="https://scale.com/nucleus">Scale Nucleus</a> is the mission control for your data, providing advanced tooling for curating, visualizing and exploring machine learning datasets and debugging ML models. We recently launched an all-new Nucleus for dataset management, complete with privacy features and updated Autotag capabilities, all to help teams build better ML models by building better datasets. Now, Nucleus can help teams manage 3D datasets to develop highly accurate perception models for a range of use cases, including autonomous vehicles, robotics, and augmented/virtual reality (AR and VR). </p>
<p>Scale Nucleus helps you visualize and debug 3D Sensor Fusion data by making it easy to:</p>
<ul>
<li>Construct Scenes that contain a sequence of LiDAR point clouds and their corresponding camera images over time</li>
<li>Visualize, query and share sensor fusion data with colleagues </li>
<li>Integrate Sensor Fusion data with your 3D model predictions to uncover and identify:<ul>
<li>Discrepancies between 3D object detection model outputs and ground truth</li>
<li>False positive, common class confusions, and cuboid predictions that have low Intersection over Union (IoU) with ground truth</li>
<li>Model performance and improvements over time</li></ul></li>
</ul>
<p>The Nucleus API makes it easy to import your data, construct scenes, and add model predictions. You can also import data from an existing Scale project directly into Nucleus.</p>
<h2 id="aprimernucleusscenes">A Primer: Nucleus Scenes</h2>
<p>Scenes are the building blocks of 3D datasets in Nucleus and are designed with point clouds as the primary unit of data. More formally, a Scene is a sequence of Frames over time, where each Frame contains exactly one point cloud and an optional set of camera images. Here we can see a LiDAR point cloud above two paired images (the full dataset includes more than just front and back cameras, but a full suite of sensor data), which helps explain why we designed Nucleus Scenes to encapsulate data across both time and sensor modalities:</p> <figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/00157f16-6ed9-4ffc-45cc-60b676806e00/public" alt="Structure Diagram of Nucleus 3D Scenes" />
        <figcaption>One useful way to interpret a LiDAR Scene is as a table of sensor readings. A row in the table corresponds to a sequence of readings from a particular sensor, while a column in the table represents a collection of sensor data from the same timestamp (we call this a “frame” of the scene).</figcaption></figure><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6ea98ea3-b4d4-40eb-f36f-3562514cab00/public" alt="Complex Nucleus 3D Scene on SF's Embarcadero, Quarter Angle View" />
        </figure><div class="markdown-module-blog"><p>On their own, point clouds can be difficult to interpret, due to lack of motion cues or associated camera data. Nucleus Scenes make sets of 3D and 2D data easier to interpret for the following reasons: first, any number of 2D camera frames can be attached to each 3D LiDAR frame; and second, Scenes link frames from the same timespan so that you can examine all frames in a linear fashion.</p>
<p>Let’s now view the same Scene in Nucleus and see if we can identify the mystery object from above:</p></div><div class="player-wrapper-blog">
    <iframe src="https://scaleai.wistia.com/embed/medias/8ikm69knof" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  </div><div class="markdown-module-blog"><p>After uploading predicted cuboids from an ML model for your Scene, Nucleus automatically computes performance metrics under the hood. You can sort predicted cuboids by IoU or filter by false positives to quickly narrow in on your model’s failure modes. (These features have been available for 2D image analysis for some time, but today we are announcing their ability for 3D data, too.)</p>
<p>To summarize, by creating and uploading 3D Scenes to Nucleus, you can easily visualize LiDAR point clouds and images side-by-side, quality-check your ground truth annotations, and search your data based on metadata or even ML-produced attributes. Lastly, you can use Nucleus to measure key prediction performance metrics, such as IoU and false positives.</p></div><figure class="fullwidth-image-blog"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6908e9df-249d-47df-4db5-a39827556c00/public" alt="Birds' Eye View of SF Embarcadero as Seen through LiDAR and Multiple Image Sensors in Nucleus Scene" />
        </figure><div class="markdown-module-blog"><h2 id="gettingstartedwith3ddatainnucleus">Getting Started with 3D Data in Nucleus</h2>
<p>Visualizing, querying, and measuring the quality of sensor fusion datasets is a challenge facing even the most sophisticated ML teams. With the newly launched support of multimodal scenes, Nucleus makes it easy to navigate through massive datasets, and quickly narrow down on the most difficult examples. Unleash the full potential of your datasets by leveraging Nucleus for debugging, search, and quality related workflows.</p>
<p>The complete code for this tutorial is available in <a href="https://colab.research.google.com/github/brwillia/nuc-3d/blob/main/nucleus_3d_demo.ipynb#scrollTo=rr88VqEHaL2_">this Colab notebook</a>. If you’d like to follow along step by step and create a personal dataset on Nucleus, you will first need to make a <a href="http://scale.com/signup">free Scale account</a> and obtain a test API key (which you can <a href="https://dashboard.scale.com/settings/profile">find here</a> if you navigate to your user icon, and then click on API keys). You can then copy the notebook and add your personal key to upload your data and try out Nucleus for yourself. At the end, you’ll also find a section that shows you how to format and upload your own LiDAR data for curation and management in Nucleus if you’re ready to manage your own 3D data in Nucleus.</p></div>

Learn how to ingest and curate 3D LiDAR point cloud data in Nucleus.

Nucleus now supports 3D Scenes.

Managing 3D Sensor Fusion Data with Scenes in Scale Nucleus

Kevin Scott, CTO of Microsoft shares how we can democratize AI for the benefit of everyone in society.

Highlights From Kevin Scott Microsoft - TransformX 2021 Conference - Scale AI

Microsoft CTO Kevin Scott's Seven Insights on the Future of AI

TransformX Conference brought together leaders, engineers, & researchers to explore the shift from research to reality within AI & Machine Learning.

Highlights From TransformX 2021 Conference - Scale AI

Highlights From TransformX 2021 Conference

<p>Scale AI is proud to announce that the Information Security Management System (ISMS) for Scale’s AI data platform is now certified to meet ISO 27001 standards. </p>
<p>As a cloud-based data platform for AI, our customers need to be confident that we securely handle their data. The <a href="https://www.iso.org/isoiec-27001-information-security.html">ISO 27001 standard</a> is a globally recognized information security standard developed and maintained by the International Organization for Standardization (<a href="https://www.iso.org/home.html">ISO</a>). </p>
<p>Our commitment to enterprise-grade security is one reason leading ML teams such as NVIDIA, iRobot, Toyota Research Institute, Open AI, and more rely on Scale to accelerate the development of their AI applications. If you’d like further details the ISO 27001 certificate, please contact us at security@scale.com or sales@scale.com</p> 

Scale AI is proud to announce the availability of our ISO 27001 certificate 

Scale AI | ISO 27001 Compliance

Our Commitment to Securing Customer Data: Scale + ISO 27001

<p>
  Today, we are thrilled to announce the
  <strong>General Availability</strong> of
  <a href="https://scale.com/rapid">Scale Rapid</a>, the fastest way to
  production-level quality labels, with no data minimums. With Scale Rapid, ML
  teams can get an initial set of high-quality training data in a matter of
  hours. We are also pleased to announce a new
  <a href="http://scale.com/university-research">University Research Program</a
  >, offering students and researchers reduced rates on Scale Rapid.
</p>
<p>
  Often, quality labeled data will be the blocker in experimenting with
  different model setups. Slow iteration speed with labeling can significantly
  slow down AI development.
</p>
<p>
  Using Scale Rapid, ML teams can stand up production-quality data pipelines in
  three simple steps: set up a project, calibrate on instructions and quality,
  then scale to production.
</p>
<p>With Scale Rapid, ML teams can:</p>
<ol>
  <li>
    <strong>Experiment quickly</strong> by setting up labeling projects in
    minutes and receiving initial datasets within hours.
  </li>
  <li>
    <strong>Iterate over potential edge cases</strong> and instructions by
    getting fast and real-time feedback on labeling instructions and ensure
    high-quality annotations.
  </li>
  <li>
    <strong>Scale to production-level pipelines in days</strong> with precision
    quality control. No need for lengthy contract negotiations, purchase order,
    or committed spend.
  </li>
</ol>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8876b33d-d9c6-41e8-f868-4f8c485c6200/public"
    alt="Scale Rapid - Grata Quote"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    Scale Rapid has already annotated more than 750k image and text tasks during
    the Early Access period with usage doubling month over month. We look
    forward to improving more ML teams’ iteration speed by providing extremely
    fast turnarounds for high-quality data.
  </p>
  <p>
    Stop waiting for your labels and
    <a href="https://dashboard.scale.com/signup?product=rapid"
      >get started with Scale Rapid</a
    >
    today. For university rates, apply
    <a href="http://scale.com/university-research">here</a>.
  </p>
</div>


Stop Waiting for Your Labels. Scale Rapid is the fastest path to production-quality labels with no data minimums. 

Stop Waiting for Your Labels with Scale Rapid

Scale Rapid: Now Generally Available

<p>
  It is undeniable that high-quality labeled datasets play a crucial role in the
  development of new deep learning algorithms. However, the world's
  understanding of ML and deep learning remains in its early days. That’s why
  Scale's applied ML and ML research teams work to better understand the latest
  in ML research and how we may overcome the biggest challenges facing AI
  development today – both for our customers and the industry at large.
</p>
<p>
  Recently, our team of researchers conducted an in-depth analysis of the state
  of data in computer vision. The research paper, which was accepted to the
  <a href="https://www.icml-hill.com/"
    >Human-in-the-Loop Learning Workshop at ICML 2021</a
  >, found that high-quality annotations remain indispensable for developing
  accurate deep learning models.
</p>
<p>
  The full paper can be found
  <a href="https://arxiv.org/abs/2108.00114">here</a>. We've summarized the key
  findings below, including:
</p>
<ol>
  <li>
    the impact of dataset size and quality on model accuracy and robustness;
  </li>
  <li>the effectiveness of pretraining;</li>
  <li>the advancement of and hurdles facing synthetic datasets;</li>
  <li>self-supervised learning;</li>
  <li>continual learning and its relationship with large datasets; and</li>
  <li>the phenomenon of double descent</li>
</ol>
<p>
  Our key finding is that high-quality annotations remain indispensable for
  developing accurate deep learning models.
</p>
<h2 id="tradeoffbetweenpretrainingandfinetuningdatasetsizes">
  Tradeoff Between Pretraining and Fine-Tuning Dataset Sizes
</h2>
<p>
  Consider a fixed labeling budget. Would it be more advantageous to allocate
  funds to label a general purpose dataset, for example classification, or to
  focus on a more specific dataset, such as segmentation and object detection?
</p>
<p>
  In our survey, we found that pretraining on ImageNet does not help when
  fine-tuning on a different task, namely object detection on MSCOCO. In
  particular, we saw significant performance deterioration when using less than
  10% of the COCO dataset. But by comparing the AP@50 and AP metrics, we found
  that pretraining can help the model achieve better classification (same task
  as pretraining) but not localization (different task). If models are trained
  to saturation, they exhibit a similar level of accuracy; nonetheless, the
  pretrained model could achieve the same accuracy with fewer iterations.
</p>
<h2 id="effectiveofpretraining">Effective of Pretraining</h2>
<p>
  In this section, we survey the research using JFT-300M and the Instagram
  dataset because they are among the largest datasets. The JFT-300M contains 300
  million images with 375 million noisy labels across 18,291 classes. The
  Instagram dataset contains 3.5 billion images with 17,000 classes and unknown
  amounts of label noise. The major takeaway is that if pretraining is similar
  to the target tasks, then pretraining on massive datasets unequivocally helps.
  We also observed three phenomena:
</p>
<ol>
  <li>
    The performance of the model grows as a logarithmic function of the
    pretraining dataset size.
  </li>
  <li>
    A sufficiently large model is crucial to capture the performance boost
    offered by pretraining on massive datasets.
  </li>
  <li>
    Hyperparameter during pretraining can be drastically different from training
    on target tasks.
  </li>
</ol>
<h2 id="syntheticdatasets">Synthetic Datasets</h2>
<p>
  Synthetic datasets are a promising avenue to pursue when collecting real-world
  data is impossible. However, it is well known that deep neural networks are
  sensitive to distribution change. A model trained on a synthetic dataset still
  must be fine-tuned on real data. For self-driving car applications,
  <a href="https://arxiv.org/abs/1608.02192">Richter et al.</a> found that a
  model trained with just one-third real data alongside synthetic renders from a
  video game engine could outperform training on all real data. Real-world data,
  however, was critical to achieving this performance, accounting for an
  increase of over 20 percentage points from the zero-shot synthetic data
  baseline.
</p>
<h2 id="selfsupervisedlearning">Self-Supervised Learning</h2>
<p>
  In recent works, self-supervised learning has been shown to produce excellent
  representations. Combined with a simple linear classifier or regressor, the
  representations can be used for downstream tasks with high accuracy. The
  question we aim to answer is whether self-supervision is beneficial when large
  labeled datasets exist for downstream tasks. In
  <a href="https://arxiv.org/abs/1903.12261">Hendrycks et al.</a>, it was found
  that self-supervised learning does not improve accuracy, but does enhance
  robustness against noisy labels, adversarial examples, and input corruptions.
</p>
<p>
  For NLP applications, self-supervision has been the key for the latest model
  structures, such as BERT, RoBERTa, and XLM. These models are trained to
  predict words or tokens that are masked or replaced. In computer vision,
  contrastive learning has been shown to produce a good starting point for
  classifiers. Ultimately, self-supervised learning can reduce the number of
  annotations required to train, but it is best used in tandem with fully
  supervised learning.
</p>
<h2 id="continuallearning">Continual Learning</h2>
<p>
  The autonomous vehicle industry is a prominent example of lifelong learning
  with deep learning. As vehicles collect new data, images with a certain
  failure mode can be queried from the stream of data and relabeled by humans.
  The model then can learn these failure modes and be redeployed to the fleet of
  vehicles. This cycle allows the model to continuously improve its accuracy.
</p>
<p>
  When continually training models, however, catastrophic forgetting might occur
  in which the model no longer can have high accuracy on previous data. Using
  the
  <a
    href="https://proceedings.neurips.cc/paper/2017/file/0efbe98067c6c73dba1250d2beaa81f9-Paper.pdf"
    >pseudo-rehearsal</a
  >
  method and
  <a href="https://www.pnas.org/content/114/13/3521"
    >regularization-based method</a
  >
  could alleviate catastrophic forgetting, but no current research solves this
  problem.
</p>
<h2 id="doubledescent">Double Descent</h2>
<p>
  The double descent phenomenon postulates that medium-sized models are more
  susceptible to overfitting than their smaller and larger counterparts. As
  model complexity increases, the test error will start off as a large number,
  decrease, increase, and then finally decrease again.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/faf6fd7f-878a-4ec2-63b8-e5ac2078df00/public"
    alt="Double Descent"
  />
  <figcaption>
    Image Courtesy of OpenAI: https://openai.com/blog/deep-double-descent/
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    When early stopping is used, the double descent phenomenon becomes less
    pronounced. While there are still many open questions regarding optimal
    model sizing in small data regimes, we are far from saturating on model
    capacity in massive data regimes. In general, we conclude that — given
    sufficient labeled data with little label noise — larger models trained with
    early stopping generally are more likely to outperform smaller models.
  </p>
  <h2 id="conclusion">Conclusion</h2>
  <p>
    Ultimately, training on seemingly infinite data is hard and we’re only
    beginning to understand the benefits and pitfalls. While training in this
    regime is costly and requires significant compute resources, we believe
    that, in the long run, the cost of each step will be reduced. We see promise
    in new techniques like large-scale self-supervised learning. But with where
    the research and technology stands today, human-in-the-loop learning
    including data annotation will remain indispensable to train deep learning
    models on various machine learning tasks.
  </p>
  <p>
    Our full paper can be found at
    <a href="https://arxiv.org/abs/2108.00114">Arxiv</a>.
  </p>
</div>


Research conducted by Scale AI's ML team has found that human annotations remain indispensable for deep learning models

Scale AI Research Highlight

Human Annotations Remain Indispensable for Developing Deep Learning Models

<h2 id="introductiontotaxonomycategorization">
  Introduction to Taxonomy Categorization
</h2>
<p>
  Have you ever had trouble finding an item in a grocery store? Maybe you were
  looking for hash browns in the frozen breakfast section, but it was actually
  in the frozen vegetable section. Or you were looking for soy sauce in the
  Asian food section but it was with the condiments.
</p>
<p>
  One of the biggest challenges for retailers and marketplaces is building a
  catalog by categorizing their millions of products into a complex system of
  thousands of categories, also referred to as taxonomies. In physical stores, a
  detailed categorization is required to organize store shelves logically. In
  this new age of e-commerce and digital marketplaces, proper categorization
  offers numerous benefits, including enhanced search recommendations, more
  relevant replacement suggestions, and better compliance with local and federal
  regulations.
</p>
<p>
  Taxonomy categorization is hard not only because of the countless products in
  the world that create deeply nested hierarchical categories but also because
  of the ever-changing nature of taxonomies. It is impossible to have a single
  subject matter expert who understands the entire catalog enough to categorize
  each individual item; training a team of experts is not scalable and not
  speedy enough for today’s demands. What’s more, input data is never perfect
  and missing information can make it impossible to identify where a product
  belongs in any given taxonomy.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/4ce72094-b9a0-4a87-5c0e-589c30517800/public"
    alt="Product Classification Hierarchy"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    At Scale, we have worked with leading e-commerce and marketplace companies
    to label and categorize millions of products into complex taxonomy
    hierarchies. In this blog post, we describe some of the techniques and
    lessons we’ve learned to scale these pipelines.
  </p>
  <h2 id="dynamicconsensus">Dynamic Consensus</h2>
  <p>
    For these subjective, difficult tasks, we leverage what we call
    <strong><em>dynamic consensus</em></strong
    >. With consensus, multiple Taskers are prompted independently to select an
    answer for the same task. If the Taskers agree, the probability of the
    answer being correct increases. Simultaneously, however, we assign higher
    values to the response of a Tasker with historically higher accuracy rates
    than we do for those that previously have recorded lower accuracy rates.
    Incorporating historical accuracy scores when computing consensus improves
    future accuracy.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/943501e2-5b1f-4474-487c-381741ff8700/public"
    alt="Traditional Dynamic Consensus"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    This is where the dynamic part of dynamic consensus comes into play. We
    algorithmically adjust the number of attempts we request on a task until a
    certain confidence level is met. If, for example, three Taskers have
    different responses to the same task, we would ask a fourth or even fifth
    Tasker to keep attempting that task until we have a high degree of
    confidence in the answer. This enables us to scale labeling rapidly while
    maintaining the same quality as would be found using a small team of
    well-trained subject matter experts.
  </p>
  <p>
    As the volume of products to be categorized increases, however, the number
    of nuanced tasks where Taskers do not agree also increases. We needed a way
    to reduce the number of tasks with disagreements to allow our Reviewers —
    Taskers that have been promoted as a result of their accuracy — to spend
    their time most efficiently, reviewing only the truly most difficult tasks.
  </p>
  <h2 id="mlandhumanconsensus">ML and Human Consensus</h2>
  <p>
    For some time, our machine learning (ML) team has leveraged ML-powered
    pre-labeling and automation in our computer vision annotation pipelines to
    achieve high-quality labeling. (See
    <a href="https://scale.com/blog/how-we-scale-machine-learning">this post</a>
    to learn more about how we scale ML for our computer vision pipeline).
    Pre-labeling is a common approach for hybrid machine + human labeling
    processes in which a model predicts an answer, and then humans audit or fix
    the model predictions for better quality. For discrete tasks such as
    taxonomy categorization, however, asking Taskers to confirm or change model
    predictions didn't do much to improve their efficiency. Reviewing
    categorization tasks took as much time as an attempt from scratch. Instead,
    we experimented with using model inputs as a separate judgment in our
    dynamic consensus workflow.
  </p>
  <p>
    We started by training a classifier model to predict a product’s category
    out of a list of over 4,000 categories leveraging a
    <strong><em>golden dataset</em></strong
    >. A golden dataset is a subset of data that has already been labeled and
    subjected to multiple review levels to achieve extremely high (98%+)
    quality. The model trained on the golden dataset subsequently achieved 75%
    accuracy without fine-tuning on hyperparameters or modifying model
    architecture. From there, we created a set of experiments with different
    combinations of ML + human agreement aggregation logic, such as:
  </p>
  <ul>
    <li>1 ML attempt only (with the highest confidence)</li>
    <li>
      1 human attempt + 1 ML attempt (selected based on various confidence
      thresholds)
    </li>
  </ul>
  <p>
    For each combination, we asked our highly trusted Reviewers to evaluate
    accuracy. In our experiments, we found that one human attempt plus one ML
    attempt at 0.8 confidence provided the highest level of accuracy. Combined
    with other proprietary quality levers, we are able to deliver full batches
    of data with 97%+ accuracy at the deepest category level. Such an accuracy
    rate is extremely hard to accomplish, particularly on a complex taxonomy
    with more than 4,000 nodes and four to six levels of nested labels.
    Nonetheless, we were able to achieve this quality bar, with a single human
    judgment in most cases, by incorporating the model into our pipeline.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/69c5f430-5930-4b06-bcf2-575f645db100/public"
    alt="ML + Human Dynamic Consensus"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    Combining this hybrid approach with dynamic consensus, all tasks go through
    at least one human attempt independent of model prediction. If the model
    prediction and human attempt achieve consensus given additional validations,
    the task is finalized. With this approach, we were able to achieve consensus
    on a majority of tasks and, in fact, increased accuracy compared to that of
    a human-only workflow because the errors made by our ML models were
    uncorrelated to errors made by Taskers. This means that the model and human
    Tasker are rarely incorrect in the same way, which makes agreement between
    them highly reliable.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b4c304ed-1679-41db-45b4-5fdb5abe3000/public"
    alt="Error Overlap in Data Labeling"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    For the more nuanced or tricky tasks where the model prediction and human
    attempt have lower agreements, more judgments are added until we gather
    enough agreement so that the computed confidence score of the task exceeds
    our target threshold for the project of 97%.
  </p>
  <h2 id="conclusion">Conclusion:</h2>
  <p>
    This new ML + human consensus approach greatly reduces the proportion of
    tasks with lower agreement. The approach also allows our most trusted
    experts to zero in on the most challenging problems across the entire
    dataset.
  </p>
  <p>
    While training the classifier model, we noticed that model quality depends
    on the quality of the dataset used in training. This reinforces the idea
    that a data-centric approach to ML is critical. Talk to our team today to
    understand how we can integrate this model into your data pipelines to
    increase data quality.
  </p>
</div>


Scale Audio

Scale AI | Enhancing Taxonomy Categorization with Human & ML Consensus

Enhancing Taxonomy Categorization for eCommerce, Marketplace, and Retail AI

ML & Human Consensus: The Best of Both Worlds

<p>
  It’s been an exciting journey since we shipped Scale Nucleus for the first
  time
  <a href="https://scale.com/blog/introducing-scale-nucleus"
    >just over a year ago</a
  >. We’ve helped over 100 different organizations explore, curate and improve
  their datasets and debug their machine learning (ML) models, across semantic
  segmentation, object detection, and even 3D point clouds. Today, we’re excited
  to share an all-new Nucleus, with powerful new features and a redesigned user
  interface that make it easier than ever to create better models with better
  datasets.
</p>
<p>
  To improve ML models requires first understanding where they fail. Nucleus
  helps you go beyond aggregate error metrics, and instead quickly discover
  specific patterns of failure across your model predictions and dataset labels.
  Once you’ve identified problems, Nucleus makes them easy to fix. Address bad
  labels through efficient, model-assisted data quality assurance (QA) and
  one-click labeling integration, or improve model predictions through targeted
  data curation of what to label next.
</p>
<p>
  <a href="http://scale.com/nucleus">Scale Nucleus</a> is one of the most
  intuitive and efficient ways to select the right data for labeling: random
  sampling shouldn’t be the way you select data to label. Training
  better-performing models requires a focus on edge cases, false positives, and
  the portion of minority classes or edge cases that are relevant to the
  modeling problem you’re trying to solve. And your curation process shouldn’t
  be overly time-intensive or manual.
</p>
<p>
  As we launched Nucleus, we
  <a href="https://scale.com/blog/introducing-scale-nucleus">explained</a> that
  better ML starts with understanding your data in depth. To improve production
  ML, you need to understand your models’ qualitative failure modes, fix them by
  gathering the right data, and curate diverse scenarios.
</p>
<h2 id="fromsimilaritysearchtoautotag">From Similarity Search to Autotag:</h2>
<p>
  When Scale customers look to improve ML accuracy, they find that their models
  often struggle with minority classes and edge cases, what we refer to as “the
  long tail.” It might be a QA team that highlights challenges with a
  self-driving vehicle, like the relatively rare combination of entering a
  tunnel at nighttime behind a truck. Although this cross-section accounts for
  only a small fraction of driving scenarios, their computer vision models must
  be capable of handling them just as well.
</p>
<p>
  In the below example, we’ve identified uncertain samples, and using a
  similarity search based on internal models and feature vectors, surfaced
  similar portions of the dataset for labeling or QA. This process culminates in
  our Autotag feature:
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/907a4a44-3760-4607-8981-4fd69c669700/public"
    alt="Similarity search and Autotag"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    Autotag is an incredibly efficient way to tag similar objects or scenes with
    a new class. Simply select a few similar images of the class you are trying
    to classify or create a tag for, click ‘Autotag,’ and then further refine
    your set of images with positive and negative examples. Scale Nucleus will
    then create an Autotag, an internal classification attached to the subset of
    your dataset that Scale identified as similar. You can then retrieve objects
    in your dataset based on this classification, even if labels for it don’t
    exist in your ground truth.
  </p>
  <h2 id="thesameusefulcorewithrefinedusability">
    The same useful core, with refined usability:
  </h2>
  <p>
    Nucleus focuses on several main use cases that we’ve developed in
    conjunction with our customers:
  </p>
</div>
<div class="markdown-module-blog">
  <p>
    <strong
      >Understand the strengths and weaknesses in your dataset as you identify
      ways to improve quality:</strong
    >
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3b36ea98-b2c3-42e9-c063-a4a2a400b200/public"
    alt="Understand the strengths and weaknesses in your dataset as you identify ways to improve quality."
  />
  <figcaption>
    Using the query bar at top or powerful search options at left , you can
    discover edge cases, debug model failures, and efficiently QA any part or
    your dataset.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    <strong
      >Analyze the long tail of your dataset, with collaborative label edits,
      and granular insights like Intersection over Union (IoU):</strong
    >
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/94c22661-e595-47af-3446-858a7ae4e700/public"
    alt="Analyze the long tail of your dataset, with collaborative label edits, and granular insights like Intersection over Union (IoU)."
  />
  <figcaption>
    Object view helps you assess all classes, labels, and bounding boxes in your
    dataset and make quick tweaks to a label, approve/reject the object as
    well-labeled, or send the object to be labeled.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    <strong
      >Examine metrics, failure cases, and confusion matrices, all linked to
      your underlying training data.</strong
    >
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fd250417-e7f1-42bc-8df6-f1e141092d00/public"
    alt="Examine metrics, failure cases, and confusion matrices, all linked to your underlying training data."
  />
  <figcaption>
    The Insights tab provides you with interactive class distributions,
    correlations, and confusion matrices, from which you can access the
    underlying data for each category in one click.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    Scale Nucleus was built with the explicit goal of helping ML teams improve
    their datasets to extract better and better performance out of models
    they’re training on their data. Whether you’re starting out with an
    industry-standard dataset, and then execute transfer learning with the
    addition of proprietary data, or training a model entirely on your own data,
    Nucleus helps you train your model on the data that matters.
  </p>
  <h2 id="privacymodeusingnucleuswithoutsharingsensitivedata">
    Privacy Mode: using Nucleus without sharing sensitive data
  </h2>
  <p>
    Several prospective customers asked if they could use Scale Nucleus without
    uploading their training datasets to our cloud. Accordingly, we created
    Privacy Mode in conjunction with our existing API, letting you use Nucleus
    to curate your dataset without sensitive raw data ever leaving your servers
  </p>
  <p>
    With Privacy Mode, you can submit URLs to Nucleus that link to raw data
    assets like images or point clouds, instead of transferring that data to
    Scale. These URLs may optionally be protected behind your corporate VPN or
    an IP whitelist. When you load a Nucleus web page, your browser will
    directly fetch the raw data from your servers without it ever being
    accessible to Scale. Privacy Mode even works well with similarity search and
    Autotag, as users can
    <a href="https://dashboard.scale.com/nucleus/docs/api#create-a-custom-index"
      >create custom model embedding indexes</a
    >
    for their datasets.
  </p>
  <h2 id="whatcustomersaresayingaboutnucleus">
    What customers are saying about Nucleus:
  </h2>
  <blockquote>
    <p>
      “KeepTruckin encounters all manners of surprising edge cases in real world
      data collection, so when it comes to knowing we’re labeling the most
      valuable subset of our collected data, we turn to Scale Nucleus. Its
      intuitive visualizations, query engine and Autotag help our teams improve
      both data quality and models, all in the same motion.”
    </p>
  </blockquote>
  <p>—Ali Rehan, Engineering Manager AI/Vision Products, KeepTruckin</p>
  <p>
    Nucleus makes it easy to query data samples based on their metadata, or
    simply bucket images based on “similarity,” specifically along feature
    vectors of a Scale-trained model. For example, if you know your model should
    be detecting police vehicles and it’s not doing so already, you can quickly
    query for a handful of examples, find similar images, and through simple
    positive and negative feedback to Nucleus, identify a subset of your dataset
    to send out for labeling.
  </p>
  <p>
    One year in, Scale Nucleus has already come a long way towards building a
    new generation of ML tooling, but we’re just getting started. If you’d like
    to join us in this journey and try Scale Nucleus, sign up
    <a href="http://scale.com/nucleus">on our website</a> or
    <a href="https://calendly.com/russell-kaplan/30min">schedule a demo</a>.
  </p>
</div>


Scale Nucleus helps you curate and automatically tag your training data.

Nucleus relaunched: making data and model quality even easier.

Improving Datasets and Debugging Machine Learning Models with Scale Nucleus, Rebuilt from the Ground Up 

<p>At Scale AI, customers come first. This is a core credo that drives many of our daily decisions. We take the trust of our customers and the security of their data seriously, and today, we are proud to announce the availability of our System and Organizational Controls (SOC) 2 Type II report, and the achievement of HIPAA compliance. </p>
<p>As a cloud-based data platform for AI, our customers need to be confident that we securely handle their data. SOC reports are prepared by independent Certified Public Accountant (CPA) auditors based on the internationally recognized <a href="https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/trust-services-criteria.pdf">Trust Services Criteria</a> framework developed by the American Institute of Certified Public Accountants (<a href="https://www.aicpa.org/">AICPA</a>). Achieving compliance with SOC 2 Type II requires Scale to demonstrate its compliance with the service commitments and system requirements in the Security Trust Services Category. </p>
<p>The Health Insurance Portability and Accountability Act (<a href="https://www.hhs.gov/hipaa/index.html">HIPAA</a>) created national standards to protect sensitive patient health information (PHI). Achieving compliance with the regulations set forth by HIPAA demonstrates Scale’s commitment to safeguarding PHI. Our HIPAA compliant service allows customers in the highly-regulated and security-conscious healthcare industry to utilize Scale as their secure, compliant data platform for AI.</p>
<p>Our commitment to enterprise-grade security is one reason leading ML teams such as NVIDIA, iRobot, Toyota Research Institute, Open AI, and more rely on Scale AI to accelerate the development of their AI applications. If you’d like further details on our SOC 2 Type II report or HIPAA compliance, please contact us at security@scale.com or sales@scale.com</p> 

Scale AI is proud to announce the availability of our SOC2 Type II report and achievement of HIPAA compliance 

Scale AI | SOC 2 Type II and HIPAA Compliance

Our Commitment to Securing Customer Data: Scale + SOC 2 Type II and HIPAA

<p>
  For many ML teams, acquiring high quality labeled data is a barrier to
  iterating quickly on different models. That’s why we’re thrilled to offer
  early access to <a href="https://scale.com/rapid">Scale Rapid</a>, a new
  product that enables machine learning engineers and researchers to
  <strong
    >receive high-quality labels and instruction feedback in as little as one
    hour</strong
  >.
</p>
<p>With Scale Rapid, you can:</p>
<ul>
  <li>Create your own labeling projects</li>
  <li>Upload your data labeling via UI or API</li>
  <li>Design and submit your own labeling instructions</li>
  <li>Direct quality improvements with new or updated evaluation tasks</li>
</ul>
<p>
  Scale was founded to solve the challenge of scaling data labeling pipelines to
  production-level volumes. Today, as more diverse and agile teams experiment
  with ML, those teams are all too often blocked by the inability to update and
  receive prompt feedback on labeling instructions for rapid iteration. Scale
  Rapid solves for this by providing:
</p>
<ol>
  <li>Production-quality labels at lower volumes</li>
  <li>
    More insight and control over labelling workflow through quality and
    throughput metrics, including edge case detection
  </li>
  <li>
    No minimum commitments, annual contracts, or platform fees. You can pay as
    you go, per label, not hourly rates—with a credit card, not a contract
  </li>
</ol>
<p>
  Our product architects have worked closely with some of the most seasoned deep
  learning and ML teams in the world. We quickly realized that getting quality
  labeled data, and getting it fast, is often the obstacle to getting good
  results from experimental models. In order to stay agile, research teams and
  startups alike need access to the best training data possible in the shortest
  amount of time. Scale Rapid solves these problems by using Scale’s core
  operations infrastructure and enabling an iterative feedback loop with the
  customer.
</p>
<p>
  Currently in Early Access,
  <a href="https://scale.com/rapid">Scale Rapid</a> gives customers the fastest
  way to production-quality labels with no data minimums, while offering full
  control.
</p>
<h2 id="whatcustomersaresayingaboutrapid">
  What customers are saying about Rapid:
</h2>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/73fe19dd-5b59-4bd0-d61f-eea2b5a28400/public"
    alt="Collectability Quote"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    Other research and engineering groups at Adobe, Bossanova, Grata, Square,
    and X2 AI are also labeling rightsized batches of data with Scale Rapid.
  </p>
  <p>
    If you are looking for a cost-effective data annotation solution for rapid
    experimentation, join the waitlist or contact us today for more information
    on our early access program.
  </p>
  <p>
    To learn more about Scale Rapid, watch our
    <a href="https://scale.com/events/scale-rapid">Tech Talk</a> that was held
    on August 18th, we walked through a live demo and took questions.
  </p>
  <h2 id="otherrecentproductadvances">Other recent product advances:</h2>
  <p>
    Scale Rapid isn’t our only recent product development. To expand on the
    image training needs for autonomous vehicle customers, we launched
    <a href="https://scale.com/mapping">Scale Mapping</a> to address both
    computer vision and path planning needs. Also,
    <a href="https://scale.com/nucleus">Scale Nucleus</a> helps you curate your
    dataset and select edge cases, false positives, and under-represented
    classes for labeling rather than simply randomly sampling your dataset for
    labeling.
  </p>
  <p>
    Join <a href="https://scale.com/rapid">the waitlist</a> or
    <a href="https://scale.com/sales">contact us</a> today for more information
    on our early access program.
  </p>
</div>


Get early access to Rapid, Scale’s new onramp for rapid experimentation.

Quickly Iterate on Datasets with Higher-Quality Annotations

Scale Rapid: Fast Experimentation with Production-Quality Labels

<p><strong>If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home">Join the Scale Exchange Community</a> to access these recordings and previous events.</strong></p>
<p>Scale’s mission is to accelerate the development of AI. For eCommerce and consumer marketplaces, AI powers high-quality catalog curation, user-to-product personalization, intelligent customer support, and real-time demand forecasting. Through the adoption of AI, leaders in the space have the opportunity to create tens of millions of dollars of value via better recommendations, increased conversion, and deep user satisfaction.</p>
<p>Last week we hosted Scale Converge, to showcase the latest advances in AI and ML for eCommerce and online marketplaces. As we shared in our opening remarks, “it is imperative for every industry to leverage AI, or otherwise risk being left behind.”</p>
<p>We brought together technical leaders to hear from the people and companies leading the eCommerce and retail industry forward. It was an event full of thoughtful conversations that examined the advancements made, the challenges that still lie ahead, and what we can expect in the years to come.</p>
<h2 id="sessionrecapusingaitolaunchnewproducts">Session Recap: Using AI to Launch New Products</h2>
<p>Toby Espinosa, VP at DoorDash joined Alex Wang, Founder &amp; CEO at Scale to discuss how DoorDash launches new products and services and how AI can help drive scale. Toby discussed how AI comes into play for DoorDash’s white label business and their framework for launching new products stating “We do a very simple process of, what is the problem for one of our core customers? How big is this? Is it a large enough business opportunity for us to dive head in first? And then do we have a competitive advantage? And if we do, if all those things match, it's like you're in Vegas and you see the slot machines and they go ding, ding, ding, they all match.” </p>
<p>Toby also shared his insights into hiring the best talent profiles for those who can do the “and” in our current world. Given the operational nature of DoorDash’s business, their approach to innovation and problem solving often involves putting a small team on a problem or opportunity to solve at a micro level, and then using AI and technology to scale. Thus, when searching for product talent DoorDash looks for people that can do the AND. Those who can start solving the problem in the weeds, but have a vision for scale. </p>
<h2 id="sessionrecapcombiningaiandhumaninsightstoaccelerateaiadoptioninecommerce">Session Recap: Combining AI and Human Insights to Accelerate AI Adoption in eCommerce</h2>
<p>Sebastian Barrios, VP of Technology at Mercado Libre, Pranam Kolari, Sr. Director of Engineering at Walmart Technology, and Jason Sleight, Group Tech Lead at Yelp joined Aatish Nayak, Head of Content &amp; Language for an expert panel to discuss how the field is leveraging advancements in AI/ML to accelerate the world’s move to online retail.</p>
<p>The group discussed a variety of topics including:</p>
<ul>
<li>Applications of AI/ML across eCommerce marketplaces </li>
<li>Building MLOps platforms to support eCommerce use-cases</li>
<li>Challenges in personalization &amp; discoverability for ever-changing global customer interests</li>
<li>Utilizing human-in-the-loop for high search and catalog quality</li>
</ul>
<h2 id="challengesahead">Challenges Ahead</h2>
<p>Because of the challenges posed by our ever-changing world, scaling ML from proofs of concept to production is nearly impossible without the most up-to-date data, continuous re-training of models, and human-in-the-loop to handle long-tail edge cases.</p>
<p>But it’s not enough just to have a lot of data. AI must be developed with the most accurate, high-quality data. Without it, AI systems won’t live up to its full potential and bad data can lead to less than ideal outcomes: </p>
<ul>
<li>Off-topic recommendations to users causing churn</li>
<li>Incorrect product catalogs</li>
<li>Slow customer support cycles</li>
<li>Fragility to changing global circumstances</li>
</ul>
<p>Yet it is imperative for every industry to leverage AI, or otherwise risk being left behind.</p>
<p>High-quality ground truth data must be the foundation of every AI strategy and it is important that companies have a trusted partner helping them build their AI models correctly from day one.</p>
<p>Scale serves as that trusted partner, providing the critical data infrastructure for AI. We work on the most ambitious AI projects in the world to accelerate the development of this technology - from eCommerce and finance to transportation, robotics, and government - for companies like Instacart, Etsy, PayPal, OpenAI, Brex, and Flexport.</p>
<h2 id="whatsnext">What’s Next</h2>
<p>The continued advancements in AI have created incredible applications in today’s world but there is an exciting future that awaits. The AI revolution is inevitable – it’s imperative we build it right. At Scale, we’re excited to have a hand in creating trusted, ubiquitous use of AI.</p>
<p>We are also grateful for all our incredible speakers representing organizations including DoorDash, Mercado Libre, Walmart Technology, and Yelp.</p>
<p>If you were not able to join Converge live, the fireside chat and expert panel are now available for on-demand viewing. <a href="https://exchange.scale.com/scale/home">Join the Scale Exchange Community</a> to access these recordings and previous events.</p> 

Scale AI Hosts Scale Converge, The Future of eCommerce & Retail with AI

Scale Converge 2021 | The Future of AI

Scale Converge: The Future of eCommerce & Retail with AI

<p>
  Many AI systems rely on supervised learning methods in which neural networks
  train on labeled data. The challenge with supervised methods is getting models
  to perform well on examples not adequately represented in the training
  dataset. Typically, as the frequency of a particular category decreases, so
  does average model performance on this category (see Figure 1). It is often
  difficult and costly to achieve strong performance on the rare edge cases that
  make up the long tail of a data distribution. In this blog post, we’ll take a
  deeper look at how sophisticated data curation tools can help machine learning
  teams target their experiments toward taming the long tail.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fa8816c4-0552-489a-3131-4577f439ab00/public"
    alt="Figure 1: Long Tail Distribution"
  />
  <figcaption>
    Figure 1: This type of distribution, in which there are a few common
    categories followed by many rare categories, is called a long tail
    distribution. In the majority of deep learning applications, datasets
    collected in the real world tend to have this long-tail shape.
  </figcaption>
</figure>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/703dc6dd-ba3f-4caf-380b-d77dbffb8800/public"
    alt="Figure 2: Ubiquity of Long Tail Distributions"
  />
  <figcaption>
    Figure 2: Long tail distributions occur frequently in the real world. For
    example, the frequency of words in English writing follows a long-tail
    distribution.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    The ugly truth is that almost all AI problems worth solving are made
    difficult by the challenge of a long-tail. Imagine you’re an engineer at an
    autonomous vehicle company, working on an object detection model trained on
    image data captured from vehicles on the road. This real-world data provides
    a textbook long-tail distribution: There are many thousands of images of
    cars on the highway, but very few depicting bicycles at night (see Figure
    3). In a preliminary experiment, the subsequent model performs badly when
    localizing cyclists at night (see Figure 4). Neglecting this rare scenario
    is likely to result in high-severity errors during testing, but it’s
    critical for the algorithm to detect cyclists properly in order to prevent
    collisions. What’s the best method to address this model failure in a
    targeted way? There are many approaches to improving performance, including
    experimenting with new architectures or additional hyperparameter tuning.
    But if the goal is to improve model performance on this specific edge case,
    the most targeted approach is to add more examples of cyclists to the
    training dataset. The problem is that unlabeled data is inherently difficult
    to search; sampling randomly, one is unlikely to find more cyclists at
    night, and it’s too expensive to simply label all collected data.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e7bb4d6-f3cb-4206-5f23-a30613bac500/public"
    alt="Figure 3: Berkeley Deep Drive Class Distribution"
  />
  <figcaption>
    Figure 3: In the Berkeley Deep Drive dataset, the category “rider” occurs
    infrequently compared to the category “car.” It’s no surprise that when we
    view individual examples of riders, our model trained on Berkeley Deep Drive
    does not do a good job of localizing these objects.
  </figcaption>
</figure>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/01a950ec-c9ea-4901-65d7-befbc8d51900/public"
    alt="Figure 4: Examples of poor localization for the class “rider”"
  />
  <figcaption>
    Figure 4: Object detection predictions from an EfficientDet D7 architecture
    trained on Berkeley Deep Drive. Examples of poor localization for the class
    “rider”, a long tail class in the training dataset.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    To improve performance on the long tail of edge cases, machine learning
    practitioners can get caught up in an endless cycle of collecting more
    training data, subsampling, and retraining the model. While this type of
    experimentation is essential to machine learning development, it is
    incredibly costly with respect to time, compute, and data labeling. The cost
    of improving performance on the long tail often threatens the economics of
    AI products.
    <a
      href="https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/"
      >This blog post</a
    >
    from Andreesen Horowitz details how AI businesses seldom have the attractive
    economic properties of traditional software businesses. Where software
    products benefit from economies of scale, AI businesses experience the
    opposite; as model performance improves over time, the marginal cost of
    improvement increases exponentially. It may require ten times the initial
    training data to yield any significant model improvement. Given that
    expensive model retraining is unavoidable, machine learning teams should
    focus on making iterative experimentation as efficient as possible.
  </p>
  <p>
    One way to focus experiments on improving the long tail is to use model
    failures to identify gaps in the training dataset and then source additional
    data to fill those gaps. Think of this approach to machine learning
    experimentation as “mining the long tail.” With each experiment, identify a
    failure case, find more examples of this rare scenario, add new examples to
    the training dataset, and repeat.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b271a31-9926-4679-4e58-29e90178b500/public"
    alt="Figure 5: ML Development Lifecycle"
  />
  <figcaption>
    Figure 5: The ML development lifecycle consists of collecting, generating,
    annotating, managing or curating, training, and evaluating. The data
    curation or management step has the biggest impact on an experiment’s
    success, yet is most overlooked in ML discourse.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    It’s difficult to target experiments in the most under-represented scenarios
    without sophisticated tools for dataset curation. There are three
    operationally critical capabilities for most AI teams:
  </p>
  <ol>
    <li>Identify long-tail scenarios during model evaluation</li>
    <li>Reliably and repeatedly source similar examples from unlabeled data</li>
    <li>Add these examples to a labeled training dataset.</li>
  </ol>
  <p>
    Some of the most effective teams we work with at Scale face the challenge of
    the long tail head-on by building their tools, operations, and workflows
    proactively around an iterative process of data curation. Hussein Mehanna,
    VP &amp; Head of AI/ML at Cruise, highlighted this at Scale’s Transform
    conference earlier this year: “You have to have a robust, fast, continuous
    learning cycle, so when you find something that you haven't seen before, you
    add it quickly to your data, and you learn from it, and you put it back into
    the model.” This idea of targeted dataset curation is simple; in practice,
    however, it’s difficult to achieve because of the substantial engineering
    work required to index millions of raw images and surface difficult
    scenarios automatically during model evaluation.
  </p>
  <p>
    Let’s put ourselves back in the shoes of an ML engineer. Having diagnosed
    poor performance on cyclists at night, we now want to target the next
    experiment at improving this scenario. With a complete set of data curation
    tools, this workflow would be simple. From one difficult example, source
    similar images from the large set of unlabeled data and send it through the
    labeling pipeline for annotation. Once this new batch of data is annotated,
    include it in the training dataset for subsequent experiments. We call this
    data-centric approach to experimentation model-in-the-loop dataset curation.
    Not only is this a highly targeted approach for addressing long-tail
    performance but, unlike random sampling, it results in better balance across
    classes and avoids labeling redundant examples that yield little marginal
    improvement of model performance.
  </p>
</div>
<div class="player-wrapper-blog">
  <iframe
    src="https://scaleai.wistia.com/embed/medias/0o9gb9d3ss"
    frameborder="0"
    allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
    allowfullscreen
  ></iframe>
</div>
<div class="markdown-module-blog">
  <p>
    Performance on the long tail is often a make-or-break situation when it
    comes to AI in production. Especially in high-impact applications like
    healthcare — where data is difficult to collect and equity is critical —
    being proactive on edge cases is of utmost importance. Given the ubiquity of
    long-tail challenges in real-world applications, it’s surprising that there
    isn’t more discussion around how to tailor machine learning experiments
    toward these cases. ML teams should not run the risk of leaving precious
    rare scenarios undiscovered in a sea of unlabeled data. When it comes to the
    long tail, targeted dataset curation is an important tool for achieving
    performant AI models robust to the complexities of the real world.
  </p>
</div>


The success of real-world machine learning applications often depends on model performance on edge cases.

How to Tame the Long Tail in Machine Learning

<p>
  Artificial Intelligence (AI) and machine learning (ML) are increasingly being
  applied to high-stakes domains. One emergent domain involves applying ML to
  healthcare to assist doctors in providing better care for patients. In the
  United States, however, people of color face disparities in access to
  healthcare, the quality of care received, and overall health outcomes. In
  dermatology, for example, darker skin tones are underrepresented in
  dermatology residency programs, textbooks, research, and diagnoses.
</p>
<p>
  We aren’t the first researchers to investigate this problem. Academics such as
  <a
    href="https://www.mdedge.com/dermatology/article/216697/pigmentation-disorders/racial-limitations-fitzpatrick-skin-type"
    >Dr. Susan Taylor</a
  >,
  <a
    href="https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212"
    >Joy Boulamwini</a
  >, and
  <a href="https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf"
    >Timnit Gebru</a
  >
  have made great strides in identifying racial limitations around critical
  issues such as facial recognition and healthcare. In reviewing these various
  papers and studies, we recognized a need for a better understanding of the
  underlying data fueling this research.
</p>
<p>
  Specifically, we sought to understand how this underrepresentation impacts an
  ML model’s ability to accurately diagnose various skin conditions. In
  collaboration with our research partner, MIT Media Lab, we analyzed and
  evaluated deep neural networks trained on clinical images in dermatology.
</p>
<h1 id="methodology">Methodology:</h1>
<p>
  The team at Scale annotated 16,577 clinical images sourced from two
  dermatology atlases — DermaAmin and Atlas Dermatologico — with Fitzpatrick
  skin type labels. The Fitzpatrick labeling system, while not perfect, is a
  six-point scale originally developed for classifying sun reactivity of skin
  phenotype. The Fitzpatrick scale served as the basis for skin color in emojis
  and, more recently, the Fitzpatrick scale has been used in computer vision
  applications to evaluate algorithmic fairness and model accuracy. The
  annotated images represent 114 skin conditions with at least 53 images and a
  maximum of 653 images per skin condition. The annotations, which we are
  calling the Fitzpatrick 17k Dataset, have been open-sourced
  <a href="https://github.com/mattgroh/fitzpatrick17k">here</a> if you would
  like to explore the dataset in greater detail.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8176b7a8-94e7-494b-746f-17563912e200/public"
    alt="The Fitzpatrick Skin Type Scale"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    After the dataset had been labeled, researchers at MIT trained a transfer
    learning model based on a VGG-16 deep neural network architecture
    pre-trained on the seminal ImageNet dataset to classify various skin
    conditions. For more details on how the model was trained, we encourage you
    to read the full paper <a href="https://arxiv.org/abs/2104.09957">here</a>.
  </p>
  <h2 id="researchfindings">Research Findings:</h2>
  <p>
    The research found that the data used to train a model does matter. By
    tagging the images with Fitzpatrick labels, researchers found that the
    dataset contains 3.6 times more images of the two lightest Fitzpatrick skin
    types than the two darkest Fitzpatrick skin types. The underrepresentation
    of dark skin images in the dataset — and in dermatology atlases more broadly
    — led to larger disparities in the model’s ability to correctly diagnose
    skin conditions involving darker skin tones.
  </p>
  <p>
    While more research is required to identify where accuracy disparities are
    greatest across skin types, this research empirically shows that the data a
    model is trained on matters. More importantly, it matters how data is
    collected and what type of data is collected. Before developing and
    deploying large-scale ML models in the healthcare sector, we encourage
    researchers and practitioners to consider and examine biases in training
    datasets to ensure healthcare disparities are not unintentionally amplified
    by these models.
  </p>
</div>


Improving clinical dermatology diagnostics with better data

Scale AI & MIT Media Lab: Improving Clinical Dermatology Diagnostics

Scale AI and Research Partner MIT Media Lab Analyze Bias in Dermatological Datasets to Improve Diagnoses

<p>
  Have you ever missed the on-ramp for a freeway because you were in the wrong
  lane? Or had to circle a block in search of a package drop box or specific
  entrance? These moments may just be minor, albeit annoying, inconveniences for
  us as individuals. But for autonomous vehicles, ride sharing, or logistics
  companies that make thousands of daily trips and deliveries, the costs of such
  navigation errors add up quickly.
</p>
<p>
  Companies that depend on high-quality maps for precise navigation have two
  off-the-shelf solutions:
</p>
<ol>
  <li>
    <strong>Regular map applications</strong> like those used by individual
    consumers do not provide enough detail and users can’t layer their own data
    on top of these maps in a scalable way.
  </li>
  <li>
    <strong>HD map</strong> companies offer more turnkey solutions but customers
    still can’t edit or update the data themselves because the maps are owned by
    the mapping companies. Furthermore, it’s a time-consuming process to get
    maps updated and customers must wait for vendors to cover new regions.
  </li>
</ol>
<p>
  Lacking a viable off-the-shelf option, many companies opt to build their own
  maps. Scaling the process to generate large volumes of extremely precise
  annotated map data poses the same challenges as scaling data pipelines for any
  other AI application.
</p>
<h2 id="introducingscalemapping">Introducing: Scale Mapping</h2>
<p>
  <a href="https://scale.com/mapping">Scale Mapping</a> gives customers the most
  flexible, scalable, and transparent mapping solution. With Scale Mapping,
  customers can generate high-precision maps down to the centimeter level for
  simulations and real-world testing, enhance prediction and motion planning by
  effectively predicting another agent’s intent, train perception models to
  live-detect map features, and improve navigation and route planning to
  maximize efficiency.
</p>
<h3 id="howitworks">How It Works:</h3>
<p>
  High-precision annotations of large-scale maps are extremely challenging. To
  minimize cognitive load for human annotators (Taskers) and to maximize
  labeling efficiency and accuracy, Scale Mapping first breaks map sections down
  into tiles. From there Taskers start with vector annotations, or annotations
  of physical map features such as lane boundaries and crosswalks. Once the
  vector annotations are complete, semantic annotations — or the annotation of
  concepts — such as the location of package drop boxes or a bike lane becoming
  a turn lane is layered on. Lastly, relationships between map features and map
  semantics are linked, such as a traffic light to a specific traffic lane.
  After all three layers of annotations are complete, the individual map tiles
  are stitched back together to create a comprehensive map.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b4e2e6c5-8eb3-4305-9a65-afb211dbc100/public"
    alt="How Scale Mapping Works Diagram"
  />
</figure>
<div class="markdown-module-blog">
  <h2 id="realworldimpact">Real-World Impact</h2>
  <p>
    Because it can support data inputs from LiDAR sensors, cameras, and even
    existing maps, this workflow allows customers to develop, update, and own
    their own proprietary HD maps at speed. Scale Mapping is already trusted by
    leaders in the autonomous driving industry.
  </p>
  <p>
    Nuro, one of the leaders in the self-driving delivery space, and Domino’s,
    the largest pizza company in the world based on global retail sales,
    launched autonomous pizza delivery in Houston. Nuro’s R2 robot is the first
    completely autonomous, occupantless on-road delivery vehicle with regulatory
    approval from the U.S. Department of Transportation.
  </p>
</div>
<div class="player-wrapper-blog">
  <iframe
    src="https://nuro.sfo3.digitaloceanspaces.com/Nuro-Delivery-B-Roll-2021.mp4?mtime=20210411191557&focal=none"
    frameborder="0"
    allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
    allowfullscreen
  ></iframe>
</div>
<div class="markdown-module-blog">
  <p>
    Scale partnered with Nuro to improve perception and mapping models for the
    R2, starting with providing high-quality, rapidly labeled data. With Scale
    Mapping, Scale is moving up the autonomous driving stack by enhancing
    localization and mapping capabilities. Put simply, the R2 needs to know
    where it is, what’s around it, where it’s going, and how to get there. The
    data Scale provides lets the R2 do exactly that while ensuring the safe,
    accurate, and efficient delivery of pizzas.
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d156b7f7-dc7e-4f28-0442-9c4317206b00/public"
    alt="Nuro Quote"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    In terms of the impact across our Nuro projects, we doubled the amount of
    data annotated from 2020 to 2021.
  </p>
  <p>
    Talk to our team today for a demo and more information about our
    industry-leading quality with transparency, flexibility, and rapid
    turnaround.
  </p>
</div>


Scale Mapping

The flexible solution to develop and scale your own custom maps.

Introducing: Scale Mapping

Own Your Own Maps. Introducing: Scale Mapping

<p>
  Many industries that impact our day-to-day lives, including logistics,
  manufacturing, financial services, insurance, and government, still run on
  paper. These documents contain critical information that powers important
  workflows like clearing shipments past customs, processing insurance claims,
  underwriting loans, tracking machinery parts, issuing tax refunds, and parsing
  clinical lab reports. Unfortunately, document processes still involve people
  manually keying in information into digital systems. These labor-intensive
  workflows significantly increase the time it takes to extract information,
  leading to added costs, poor customer experience, and lack of scalability for
  high volumes of documents.
</p>
<p>
  To tackle this challenge, companies previously used technologies like Optical
  Character Recognition (OCR). However, OCR solutions only transcribe text
  without extracting the most relevant fields. Most “intelligent” extraction
  solutions rely on customers to manually define templates or hard-coded sets of
  rules to handle data extraction. The rules dictate which part of the document
  corresponds to a certain field, which means the system can only process the
  document layouts that fit the original template. But, if you change vendors,
  document types, or update the document in any way, then you need to manually
  define another set of rules. Solutions like these take a lot of upkeep and
  result in poor quality when dealing with complicated, variable document types.
</p>
<p>
  Imagine being able to feed any type of document into a system that can quickly
  and accurately extract fields and link entities, even from new layouts or
  formats — all without any setup or maintenance effort on your end.
</p>
<p>
  Tech-forward companies are recognizing the importance of deploying a robust
  solution for automating entity extraction and linking because high quality,
  structured data can:
</p>
<ul>
  <li>
    Improve existing services by making them faster, cheaper, and more accurate
  </li>
  <li>Enable new products based on the data unlocked.</li>
</ul>
<p>
  The challenge is building a system that can adapt to real-world variations,
  especially across semi-structured and unstructured documents without
  sacrificing accuracy. Here is how we do it.
</p>
<h2 id="scaledocumentai">Scale Document AI</h2>
<p>
  <a href="https://scale.com/document">Scale Document AI</a> relies on our
  latest technology, Adaptive AI, to deploy refined machine learning models for
  customers who demand high quality and low latency when it comes to document
  processing. We leverage base models trained on millions of data points, and
  further refine those models for each customer use case. This enables us to
  extract and link entities from highly variable documents in seconds without
  putting the burden of setup on the customer.
</p>
<p>
  What’s unique about Adaptive AI is that we deliver a solution tailored to each
  use case to extract the data our customers need at high quality — regardless
  of any changes in document layouts. Unlike existing solutions, our machine
  learning models thrive on challenging and varied documents by parsing the
  structural layout of pages, contextualizing the meaning of words, and
  understanding the relationships between different fields. We developed
  Adaptive AI to actually understand the structure and the form fields’ meaning,
  rather than simply learning where on a document to find a field (e.g.
  understanding the vendor name instead of instructing that it is usually on top
  left of document).
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7356c354-9a87-455d-d0e7-6618e6d73100/public"
    alt="Scale Document Workflow"
  />
</figure>
<div class="markdown-module-blog">
  <p>With Scale Document AI, you get:</p>
  <ul>
    <li>
      <strong>Fine-tuned document processing models</strong> trained to meet
      your exact requirements. Achieve higher accuracy than off-the-shelf
      solutions across a variety of document layouts.
    </li>
    <li>
      <strong
        >Guaranteed quality up to 99%+ and turnaround SLA times as fast as
        seconds</strong
      >
      with a fully automated or hybrid document processing workflow using our
      API. Scale provides integrated models and optional human-in-the-loop
      quality assurance workforce.
    </li>
    <li>
      <strong
        >Painless and quick setup so your teams can focus on what’s
        important.</strong
      >
      No more configuring templates or fixing errors. Upload sample documents
      and provide a list of fields that you need extracted. There is no further
      setup or maintenance required to start processing documents.
    </li>
    <li>
      <strong>Dedicated machine learning and engineering support.</strong> Work
      with a world class technical team of machine learning researchers and
      engineers dedicated to your document processing needs. We can help you
      adapt as you scale your business.
    </li>
  </ul>
</div>
<div class="markdown-module-blog">
  <h2 id="realworldimpact">Real-World Impact</h2>
  <p>
    Scale Document AI serves many industries across financial services,
    insurance, real estate, logistics, manufacturing, energy, healthcare, and
    government, across a variety of document types and use cases such as loan
    origination, customer onboarding, invoice/receipt/bill processing, fraud
    prevention, claims processing, title closing, shipment processing, and
    clinical lab report extraction.
  </p>
  <p>
    In logistics, we partnered with
    <a href="https://www.flexport.com/">Flexport</a>, which is building the
    platform for global trade, to significantly increase efficiency and reduce
    costs in what used to be a manual effort for processing critical shipping
    paperwork. We quickly deployed our Adaptive AI for important logistics
    documents like Bill of Lading, Arrival Notices, and others to reduce data
    extraction errors and improve compliance across Flexport’s global trade
    network. James Chen, Flexport CTO, explained the impact of our partnership
    at
    <a href="https://scale.com/events/transform/videos/modernizing-global-trade"
      >Scale Transform</a
    >, our conference earlier this year:
  </p>
</div>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/a186b4ed-c72c-4575-e374-aeb715927c00/public"
    alt="James_Chen_Quote"
  />
</figure>
<div class="markdown-module-blog">
  <p>
    Another applicable area for Scale Document AI is financial services,
    specifically bill pay services, loan origination, and claims processing.
    <a href="https://www.brex.com/">Brex</a>, a $7.4B startup modernizing
    banking for businesses, uses Scale Document to power its new premium bill
    pay product. When customers submit invoices in the Brex app or forward an
    email with an invoice to the designated Brex email, our Adaptive AI extracts
    all relevant information from these invoices instantly and accurately to pay
    the bill, at higher quality and faster than off the shelf solutions.
  </p>
  <p>
    We are excited to reimagine document processing to enable faster and better
    workflows for challenging document types where conventional approaches don’t
    hit the mark. If you want to achieve modern operational efficiencies through
    AI and see tangible downstream improvements in business metrics, contact us
    at <strong>documentai@scale.com</strong> or sign up on our
    <a href="https://scale.com/sales">website</a>.
  </p>
</div>


Scale Document's Adaptive AI offers customers a machine learning based document processing solution for instant results and none of the costly mistakes. 

OCR is not Enough. Try Scale Document AI.

OCR Is Not Enough. Try Scale Document AI.

<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>How did you join Scale?</h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6caad7df-a18e-4f08-af44-79c710e91600/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      I knew of Scale because one of my friends from college worked here. When I
      was choosing my first job, I was considering either joining a large tech
      company or a small startup. All my internships had been at large tech
      companies, so at the time I was nervous about whether I wanted to join a
      small startup. But I knew I wanted to do something that had a higher
      ceiling, where I could be really excited about the possibilities.
    </p>
  </div>
</div>
<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>What about the company’s mission is the most compelling to you?</h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/c8f59059-bc79-43b5-e9b9-fd1e6046bc00/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      I strongly believe in our mission to democratize and accelerate the
      development of AI. What excites me is that we're solving problems that are
      new, that other companies haven't solved before. We're tackling issues
      that are going to be more and more important as AI becomes more and more
      prevalent. So, I think the most compelling part of the mission for me is
      the impact of AI and the role that we get to play in it.
    </p>
  </div>
</div>
<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>
      You worked at big companies prior to joining Scale. What was the
      transition like? What advice do you share with others contemplating a
      similar move?
    </h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9e58c4ea-1d5c-4b79-e220-671802f4b800/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      I liked that at Scale, I was given a lot of ownership from day one. At
      larger companies, sometimes it can take years before you're really trusted
      to make large decisions, and at Scale I was given the opportunity to grow
      really quickly, to give input, and to contribute to work that I felt was
      very impactful very quickly. In terms of advice, I would say don't feel
      like you're confined to just what your job title might be. There might
      often be many different things you need to do to achieve an outcome.
      Sometimes that's engineering, sometimes that's operations, sometimes
      that's product management. And I think one of the things that has made
      Scale successful is we have a lot of people who are willing to do things
      outside of their job description to achieve important outcomes, and so I
      think that is a valuable piece of advice.
    </p>
  </div>
</div>
<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>What is the most exciting project you’ve worked on?</h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/a6e5ad69-ce2d-4300-3d13-adfd56743d00/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      What I'm most proud of is my work as the product manager for the 2D team.
      While I was working on 2D, we saw large growth to the business, in terms
      of number of customers and revenue. At the same time, we did a lot to
      improve our margins and reduce our costs. I'm very proud of that team and
      how we were able to achieve that amount of growth. It was the first time
      in my career that I was able to work on something at that level.
    </p>
  </div>
</div>
<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>What are some of your favorite things about the culture?</h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ccd5424d-7794-47b8-d69e-da9334027900/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      I really like that people are encouraged to disagree with each other and
      to have conversations about those disagreements. I think that helps make
      Scale a really great place to learn. I also really like the people at
      Scale - they’re a lot of fun to work with and have really great senses of
      humor.
    </p>
  </div>
</div>
<div class="question-answer-wrapper question">
  <img
    src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F22fc1fc060ae42dda10689e208179cfa"
    alt="Scale logo"
  />
  <div class="box">
    <h3>What is the most valuable thing you’ve learned here?</h3>
  </div>
</div>
<div class="question-answer-wrapper answer">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/12b6db19-c743-4e29-6800-0169c1a42400/public?h=250"
    alt="Sara Xiang"
  />
  <div class="box">
    <p>
      Problem-solving skills and the ability to own a business outcome. And
      there's not one specific skill that's necessary for that, but I think
      gaining confidence in my ability to do that, and gaining experience in
      doing that has been very valuable.
    </p>
  </div>
</div>
<div class="cta-wrapper center">
  <a class="cta" href="https://scale.ai/jobs">Explore Open Positions</a>
</div>


Scale Product Manager Sara Xiang shares insights on why she joined, projects she's worked on and life at Scale.

Life at Scale AI | Sara Xiang

<p>
  Scale AI was founded on the belief that better data → better AI. There has
  recently been a renewed focus on “good” data and its impact on the quality of
  models in the industry. Andrew Ng, who spoke at our first Transform conference
  earlier this year, emphasized this point: “We spend a lot of time sourcing and
  preparing high-quality data before we train the model, just like a chef would
  spend a lot of time to source and prepare high-quality ingredients before they
  cook a meal. I feel like a lot of the emphasis of the work in AI should shift
  to systematic data preparation.”
</p>
<p>
  While we have intuitively understood the importance of data quality, the
  Google Research Team paper
  <a
    href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf"
    >"Everyone wants to do the model work, not the data work’: Data Cascades in
    High-Stakes AI"</a
  >
  published earlier this year was the first we’ve come across that attempts to
  quantify and provide empirical evidence of “bad” data’s impacts.
</p>
<h3 id="overviewofthepaper">Overview of the Paper</h3>
<p>
  AI models increasingly are being applied in a broad range of domains from
  healthcare to climate change. Based on interviews conducted with 53 global AI
  practitioners working on a variety of applications, the paper’s researchers
  define, identify causes of, and present empirical evidence on the problem of
  data cascades. Data cascades are compounding events caused by data issues that
  lead to negative, downstream effects, resulting in the accumulation of tech
  debt over time. For more details on the paper’s findings, we encourage you to
  read the full research
  <a
    href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf"
    >here</a
  >.
</p>
<h3 id="keytakeaways">Key Takeaways</h3>
<p>
  The paper notes that <strong>92% of AI practitioners</strong> interviewed
  experienced at least one data cascade or adverse outcomes caused by problems
  in a data processing pipeline. The paper also cites several causes of
  cascades, but the number one cause — cited by 54.7% of the practitioners
  interviewed — was “interacting with physical world bitterness.” In other
  words, when AI systems move from their well-defined, static development
  environments and are deployed in the physical world, the systems are
  challenged by the dynamic changes inherent in the real world.
</p>
<figure class="fullwidth-image-blog">
  <img
    src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5e7113c0-9478-4cf4-54d3-839083523d00/public"
    alt="Data Cascades Causes"
  />
  <figcaption>
    Table 1: Prevalence and distribution of data cascades, triggers, and
    impacts. IN is short for India, EA & WA for East African and West African
    countries respectively, and US is for the United States.
  </figcaption>
</figure>
<div class="markdown-module-blog">
  <p>
    Many ML researchers and engineers today are taught to develop models using
    pristine datasets in academic settings. In real-world applications, however,
    the data ML engineers must work with is messy or even non-existent. While
    the paper’s focus is on high-stakes domains with limited resources, even in
    high-resource settings, ML engineers or data scientists will hack together a
    data pipeline leveraging whatever tools and services they can find. The lack
    of unified infrastructure to support comprehensive data operations creates
    structural failures that ultimately cause data cascades and their
    commensurate tech debt.
  </p>
  <p>
    Exacerbating the issue, as noted by
    <a
      href="https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf"
      >Sculley et al., 2015</a
    >, complex models erode abstraction boundaries for ML systems. Developing
    and maintaining traditional code already results in tech debt. But that is
    significantly compounded by ML-specific issues — including data
    dependencies, analysis debt, and configuration debt — that are much harder
    to detect, diagnose, and fix. The longer tech debt accumulates, the more
    pervasive it becomes. The more pervasive the tech debt, the more time and
    effort ML teams ultimately must spend to resolve issues, resulting in
    productivity loss and slower deployments.
  </p>
</div>
<div class="markdown-module-blog">
  <h1 id="wherescalecomesinminimizingdatacascadesbyfocusingondatawork">
    Where Scale Comes In: Minimizing Data Cascades by Focusing on Data Work
  </h1>
  <p>
    Scale was founded on the belief that for AI and ML, data is the new code. By
    providing a data-centric, end-to-end solution to manage the entire ML
    lifecycle, we bridge that skill gap, so ML engineers don’t have to change
    how they work. By focusing on the data, we enable our customers to focus on
    the model work, not the data work, minimizing the risk of data cascades and
    making AI systems safer and more robust.
  </p>
  <h2 id="dataannotation">Data Annotation</h2>
  <p>
    We started this journey by focusing on data annotation. When we first began
    providing data annotation solutions to our autonomous vehicle customers, our
    quality assurance (QA) systems were highly manual. Human QA Agents or Queue
    Managers would audit and re-check sample tasks and manually assign scores to
    individual Taskers. Over the years, as we expanded into new use cases and
    new industries, we have built out a variety of internal tools to automate
    many of these QA processes including:
  </p>
  <h4 id="automatedbenchmarks">Automated Benchmarks</h4>
  <p>
    With benchmarks, we programmatically take an annotation task we know is
    correct, create an error in that task (usually a common mistake we’ve seen
    Taskers make), and serve them as regular tasks to our Taskers. By leveraging
    benchmark tasks, we can automatically give Taskers scores based on whether
    they fixed the error before submitting. This then allows us to understand
    how well each Tasker is performing without manually scoring each individual
    on a variety of tasks.
  </p>
  <h4 id="linters">Linters</h4>
  <p>
    Linters are tools designed to catch errors and flag anomalies. At Scale, we
    leverage a variety of types of linters. For example, a response-based linter
    looks at a submitted task and checks for possible errors such as annotations
    that should not exist, like duplicate bounding boxes for the same object.
    Over time, we have layered response-based linters into the UI to flag issues
    for our Taskers as they annotate -- for example, warnings when the size of a
    bounding box appears too large.
  </p>
  <p>
    Moving beyond our core expertise in data annotation, our team has been hard
    at work building products to support the entire ML lifecycle. These include:
  </p>
  <h2 id="datacollection">Data Collection</h2>
  <p>
    Collecting diverse and representative data, which is the first step in
    developing a machine learning model, can be a challenge, especially for
    content and language use cases where there is much more subjectivity.
    Leveraging our existing global Tasker base, our text and audio collection
    workflows are API supported and seamlessly integrate with our data labeling
    pipeline for seamless end-to-end dataset creation.
  </p>
</div>
<div class="player-wrapper-blog">
  <iframe
    src="https://scaleai.wistia.com/embed/medias/uq2zsk4cy0"
    frameborder="0"
    allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
    allowfullscreen
  ></iframe>
</div>
<div class="markdown-module-blog">
  <h2 id="datamanagement">Data Management</h2>
  <p>
    While more training data is generally better than less data, once data is
    collected, ML teams need to select what data to annotate. Today, most ML
    teams do this either by randomly selecting data to label, or manually
    parsing through their data to identify valuable examples. Scale Nucleus uses
    active learning and advanced querying to help ML teams visualize their data,
    understand over- or under-represented labels and attributes to help mitigate
    bias, and identify edge cases to address the long-tail challenges of machine
    learning development. Nucleus also enables model-assisted QA, using model
    predictions (model errors in particular) to identify errors in labeled data
    and to improve data quality.
  </p>
  <p>
    In the video below, we show how active learning powers the Autotag feature
    of Nucleus to allow users to automatically search for and tag similar images
    — in this case flowers.
  </p>
</div>
<div class="player-wrapper-blog">
  <iframe
    src="https://scaleai.wistia.com/embed/medias/yocc96733p"
    frameborder="0"
    allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
    allowfullscreen
  ></iframe>
</div>
<div class="markdown-module-blog">
  <h2 id="datageneration">Data Generation</h2>
  <p>
    The ability to understand how ML models will react to rare or dangerous
    real-world scenarios is crucial for safety-critical computer vision
    applications. However, it’s not always feasible to collect enough examples
    of edge cases in the real world. Currently in alpha, Scale customers can
    augment ground-truth training data or generate scenarios from scratch to
    simulate a variety of cameras, sensors, or environmental conditions such as
    fog or night-time lighting. With data generation, ML teams can synthesize
    specific, troublesome scenarios (often pre-labeled) to expose their models
    to more edge cases and under-represented scenarios than they could otherwise
    collect.
  </p>
  <h1 id="realworldimpact">Real-World Impact</h1>
  <p>
    As we noted in our
    <a href="https://scale.com/blog/series-e">Series E Announcement</a>, we’ve
    already seen the impact that the right data infrastructure can bring to our
    customers. From 10X’ing throughput for Toyota to improving States Title’s
    document processing model to more than 95%, we have freed up many of our
    customers’ ML teams to focus on the model work instead of the data work.
    We’re proud to be trusted by the world’s most ambitious AI teams to build
    the data foundation that will set up their organizations for success for
    decades to come.
  </p>
  <p>
    In the coming weeks, we’ll also dive deeper into some of the topics — like
    how to solve for the long-tail distributions of data with better data
    management — on our blog. In the meantime, if you or your team are looking
    for an end-to-end solution for your datasets,
    <a href="https://scale.com/sales">reach out to us</a>. We’d love to hear
    from you.
  </p>
</div>


Scale Collect

Better Data = Better AI. Understanding the downstream impact of "bad" data and how to mitigate these issues.

Products

Solutions

Resources

Company

Events