Jared Glover, CEO of 3D vision and motion planning software company CapSen Robotics joins Jimmy Carroll, VP of Operations at TECH B2B Marketing on the Manufacturing Matters podcast to discuss the latest in 3D vision-guided bin picking/pick-and-place applications.
CapSen Robotics deploys 3D cameras and proprietary 3D alignment and machine learning algorithms to detect the positions and orientations of multiple objects in cluttered scenes. The company’s CapSen PiC product is a general-purpose 3D object instance detection system that can turn any industrial robot arm into a bin picking work cell.
During the podcast, Glover discusses the latest trends in 3D vision-guided robotics, the use of AI in robotics, and industrial automation overall, while also diving into some of the difficult applications that CapSen’s system has solved, the different industries and applications its being deployed into today, and more.
Jimmy Carroll: [00:00:06] Hello everybody. My name is Jimmy Carroll. I’m the vice president of operations at Tech B2B Marketing. Thanks for joining me on the Manufacturing Matters podcast. Today I have the pleasure of being joined by Jared Glover, who is the CEO of CapSen Robotics. Jared, welcome and thank you for taking the time today. Really appreciate it.
Jared Glover: [00:00:22] Yeah. Thanks for having me, Jimmy.
Jimmy Carroll: [00:00:24] Of course. So for those who are unfamiliar, can you please explain what your company does and how it started?
Jared Glover: [00:00:31] Yeah. So we make software that gives robot arms more capabilities for handling clutter. So picking things out of bins, picking objects off of shelves, stacking, destacking them, machine tending. Our software uses the latest techniques in AI, computer vision, motion planning control, and enables those kind of new applications. And then in terms of how we started, I’ve been doing research in robotics for a couple of decades. I started at Carnegie Mellon and then went to MIT for my PhD, and I did research in some of the technologies, the 3D vision in particular. And also on the motion planning side, I worked on a robot that learned to play ping-pong, which was a really fun, fun, high-speed project. So that just kind of naturally led into thinking about how it could be used in industry, and then I pulled in my partner that I started the company with, Mark. I pulled him in. I had worked for him between my master’s and PhD at a finance company in New York. And he’s been managing software teams for decades. Already before we started the company, he’d been doing that, I brought him in because I knew that a PhD like me needed help to build really industrial-strength technology. And so that’s how we started.
Jimmy Carroll: [00:02:02] Interesting. So when think of bin-picking applications, I think of those systems that typically involve the use of 3D machine vision, a robotic manipulator, and some sort of software. But bin picking is obviously much more than just vision and robots. So to you, Jared, what’s the key to successful implementation of a bin-picking system?
Jared Glover: [00:02:21] So I think there’s really two things. The software has to do more than just bin picking typically. Typically, the task for the robot isn’t just to get an object out of a pile, get an object out of the bin, but it’s a machine-tending application. It has to insert the part into a machine in a particular way or it has to assemble it with some other part or it has to package it or what have you. Palletize. And so the software has to be able to reason about all the steps of the task, not just the bin-picking task typically. What we focus on is giving the software all those additional capabilities. And from the user’s perspective, from the customer’s perspective, you really want to be working with a company, a technology company that can provide the whole solution, that can solve the whole application. And whether that’s a prefab cell where they’ve done it a hundred times already, they’ve done exactly that application 100 times, or whether it’s a new application but where you have a partner, you have a vendor that is solving the whole problem, not just giving you a robot and a piece of software and you have to figure out the rest, including all the end effectors and fingers and all of that, which can be quite tricky sometimes. You just want to want a partner that can solve the whole thing for you.
Jimmy Carroll: [00:03:45] So ultimately finding a partner that can task a robot to pick a part in a cluttered scene — that presents a challenge. And many companies have struggled with that. So companies that have struggled with bin picking, how do they overcome this challenge?
Jared Glover: [00:03:59] Yeah, it’s funny. That’s a very typical customer for us. We often get customers that have tried to solve some bin-picking-related task, either on their own, just using tools that are available off the shelf, the software tools that are available off the shelf, or even with another partner, another robot company that has more traditional tools for doing bin picking. So that’s a very typical story, that they’ve been trying for a while and haven’t been successful, and then they come to us and we’re able to solve things. But some common problems are, maybe they have no touch areas on the objects that they’re trying to pick, or maybe they need to clear the bin and they weren’t able to do that before. Another really common one, like I mentioned a minute ago, is what are you doing with the part after you pick it up? Are you able to know how you’re holding it well enough and regrasp it if you need to so that when you insert it into the machine, you’re sticking the right end of the part into the machine 100% of the time? Does it have to be like an inspection step afterwards to make sure that you’re holding things in the right way or make sure you’re placing things in the right way? What kind of motion constraints are there? Are you allowed to turn it upside down as you’re moving it from point A to point B? Or do you have to keep it in a certain or fixed orientation? All those kind of things are common issues. And again, that’s why we’ve worked so hard to make sure that we can solve the entire application for customers.
Jimmy Carroll: [00:05:35] Yeah. So one of the things that you mentioned — I wanted to ask about partnerships. Partnerships are so important in this space. CapSen — I’m talking to you in the past, Jared. I think that maybe in the past some people have considered you to be an integrator, but you’re not an integrator necessarily. How do you partner with both integrators and other robotics companies?
Jared Glover: [00:05:56] So it’s interesting. The first bin-picking application we installed was picking hooks out of a bin for a wire and spring company here in Pittsburgh. Very complicated application. The hooks were entangled. They had to be disentangled and then regrasped so you could put the right end into a machine. And it’s a very precise application. And we ended up doing the full integration on that project to get the experience, to know what had to be done, not because we wanted to be an integrator but so we’d eat our own dog food, so to say, and really understand what it was that integrators were going to need from us. But the plan all along and immediately after that, we said, all right, now we’re going to work with integrators. We don’t want to do all of the electrical work, all of the fab work and all of that. We want to provide them with the tools they need and the support, the services as well, to make our software enable these applications without being the integrator. And so what we found is that even with integrators that have lots of capabilities, lots of technical staff that are excellent at programming robots, they are still very happy to have us program the bin-picking part of the whole system . . .
Jared Glover: [00:07:20] . . . and sometimes get involved with end effector design and small hardware, mechanical issues that come up with the whole system, again because there’s so much interplay between the different parts of the system. For a bin-picking application, you really want the technology provider like us to solve the whole application, which involves mechanical and software components. Now the next stage of our evolution, though, is going to be taking what we’ve learned in solving these whole applications for every single install with our integrator partners and providing the same tools we use to do all the configuration and programming and end effector design, providing those as well to our integrator partners so that they can start to do the entire programming and configuration on their own. But so far, again — just the whole practice of eating your own dog food — we wanted to get really good at doing that ourselves and make sure that our tools are very robust and easy to use before we try to get our partners to do all that heavy lifting. But that’s the evolution that we’re in the process of.
Jimmy Carroll: [00:08:38] Now I know you also work with some robotics companies. How do you how do you partner with them typically?
Jared Glover: [00:08:45] With robotics companies, it’s more of a sales and marketing partnership typically. So we make joint marketing videos together and advertise on each other’s websites. We’re going to be in one of our robot partner’s booths at Automate in May. Denso. And actually probably in Calvary’s booth as well, one of our integrator partners. But, yeah, it’s mostly a marketing partnership with them. Now that being said, we support many different robot brands, many different camera brands, end effector brands. So our software itself is hardware agnostic, which is useful for the end customer that may have their own preferences or standards. But we definitely work closely with our robot partners on the marketing side.
Jimmy Carroll: [00:09:33] Okay, great. So let me circle back to the initial question before I got a bit sidetracked there. When it comes to CapSen and their approach to bin picking in a cluttered scene, what approach do you take?
Jared Glover: [00:09:46] It really depends on the application. As far as the overall approach, there’s a lot of different tools we have in our software tool kit and hardware tool kit, but there are some times where all the parts are rigid and they’re exactly the same every time, and so you can use the full 3D model of the object and do 3D model matching, what’s called CAD matching oftentimes, to understand exactly where the parts are in the bin or on the shelf and how to go and pick them up and manipulate them. There’s other times where maybe the part is rigid and it’s the same shape every time, but it’s extremely, extremely shiny or transparent. And so you can’t use a 3D camera to get a good point cloud to get a 3D observation of the bin. And so you have to use more machine learning, deep learning techniques, that sort of thing to get the job done. So it really depends on the application. There’s others where it’s a sorting application. So you’re picking something up without knowing what it is, and then you’re holding it up in front of another camera. Maybe you’re reading barcodes to help you differentiate. So the approach is different depending on the application. But generally speaking, what we have worked the hardest on in our software is using geometric reasoning and planning. And so whether that means you’ve got the object geometry or not, we have geometry models of everything in the environment. And even if you don’t know the object’s geometry ahead of time, once you pick it up, you’re looking at it so that you know exactly what it’s shaped and how you’re holding it. And then we’re using all of that geometry information about the environment and the task so that our software can plan the motions for the robot to accomplish the task.
Jimmy Carroll: [00:11:41] Yeah. So you mentioned software, and that’s where I was going to go next. So this is good. I mean obviously, bin picking involves more than just robots and 3D vision. You need software that can segment and identify an individual object and find that so that the robot can make a pick. What else can you talk about in terms of the software? Maybe for those who are less familiar — like the role and the importance of the software, which is kind of the heart of the system?
Jared Glover: [00:12:05] Yeah, so I’ve touched on a couple of them already, but there are four main software components in our system and in any 3D-vision-guided robotics software that’s out there, has to have these four components. So there’s the vision software, so the robot can look at what’s in the bin or what’s around it and understand exactly where things are. Then there’s motion planning software, because the parts are jumbled and in different configurations every time. So the robot has to move differently every time to pick them up and do things with them. And the motion planning software goes beyond just figuring out how to move the robot arm from point A to point B without colliding with things. You’re talking about, okay, how do I, where do I grasp on this part so that I’m not colliding with other objects in the bin? Which object do I pick in the first place? Which orientation do I pick the part in? Do I need to put it down and regrasp it again to do what I need to do next? Do I need to disentangle it? Is it going to get tangled up with another part? Do I need to separate it somehow? Do I need to place it in a pattern or stack it or something? So all those things are part of the motion planning software. Then that’s just planning what to do. Then it actually has to execute that plan on the robot. And so there’s an execution and control component to the software which executes the motions and communicates with other devices, PLCs, sensors, and so on and decides what to do in real time. Maybe there’s some interrupts that say, hey, if you feel some force you don’t expect at this point in the motion, how do you recover from that? Maybe there’s another way you can try to accomplish the same thing or maybe you just give up and get a user to come over.
Jared Glover: [00:14:02] So there’s a lot of sophistication in the execution and control. And also can you do multiple things at the same time? Can you plan for the next motion? You take the image and plan for the next motion while you’re still doing the previous pick-and-place task. So a lot of things that happen in parallel as well in the execution and control that give you the high throughput and fast cycle times on these things. And then the final, the fourth component is the user interface, which is all about not just how do you how do you control the system, how does the user control the system and see what it’s doing, but it’s how do you configure the robot in the first place? How do you program it and teach it? How do you teach it about the new parts, the new application, the environment. All of that — you have to have very sophisticated tools in the user interface to enable the configuration of the system without writing new C++ code for every new application. That’s something we’ve worked very hard on is to make sure that we’re not writing new C++ code to handle a new part or to take the robot to a new environment. It’s all done in configuration and the user interface and teaching that can be done by technicians that don’t know how to program.
Jimmy Carroll: [00:15:26] So related to that question is something you touched on earlier, and it’s the topic of AI. So these days, many companies are touting so-called AI-based bin picking, but maybe the term is misleading. Maybe it’s more like a bin-picking system that incorporates AI algorithms for certain functions. So how do these AI algorithms help in picking applications, both in terms of CapSen and the greater marketplace?
Jared Glover: [00:15:51] So the way that AI is used, the term has evolved over the years. These days I think it’s mostly referring to chat bots and other things like that, or image generation. But really artificial intelligence encompasses pretty much anything sophisticated that computers and robots have done for the last 30 or 40 years, starting with computers that play chess and search algorithms all the way up to machine learning, optimization, motion planning. Everything that we’ve talked about today on the software side is one type of AI or another. Now there are some types that people talk about more often these days when they’re referring to AI, deep learning, the generative stuff to generate text and images, all of that. In a lot of that, deep learning is used again as a tool, one of the tools that we use — not the only tool — but it’s one of the tools that we have that enables certain applications, like those very shiny parts or transparent parts, that sort of thing. But looking at some of the more recent tools like the chat bots, that type of AI is not going to solve the problems that we’re working on anytime soon. It’s not going to suddenly give robots human-level capabilities and adaptability. There’s a big difference between having a conversation that sounds sort of reasonable and actually moving around and touching things in the real world and doing assemblies and machine tending and all of that. There’s a big difference. So that type of AI is not solving anything for us yet unfortunately.
Jimmy Carroll: [00:17:39] It’s sort of like a hype versus reality thing, right? So in reality, in the space that you live in, what strengths do these AI, machine learning, deep learning algorithms offer in bin picking.
Jared Glover: [00:17:50] So again, if you look at the umbrella use of the term AI, which is really what it’s meant for decades, you can’t do bin-picking-type tasks without AI. You have to do motion planning, you have to do optimization and machine learning to solve bin picking. There’s no other way. Now if you’re specifically talking about deep learning, for example, like I said before, there’s some parts where deep learning is definitely necessary and there’s others where it’s not, and there’s some where you could solve it with deep learning or with a more classical geometric-based algorithm. And there’s a lot where you want to combine them both. We’re never just using deep learning. I think there’s a couple of companies out there that claim maybe they’re trying to train end-to-end deep reinforcement learning systems. But typically, when you really talk to their engineers, they’ll say, oh, well, yeah, we do that for the first prototype, but to actually install it, we had to add all this other stuff, the classical stuff, to make it safe and to verify what it’s doing is correct and all of that. So, yes, eventually maybe these deep reinforcement learning systems that are trained end to end on a task, they could get sophisticated enough. But you know we’re a long ways. There’s a lot of basic research that has to happen. It’ll probably be after I retire that we’re really talking about not having to program robots at all, that they just learn to accomplish a task on their own.
Jimmy Carroll: [00:19:30] Yeah. So there’s a lot of interesting stuff there. I’m just thinking along the same lines, there are companies in the marketplace today that may go as far as to say that something like lighting doesn’t matter, which probably isn’t true. So in terms of the role of consistent lighting in robotic pick and place or bin picking, can you talk about that a little bit?
Jared Glover: [00:19:54] Yeah, it completely depends on the object material. So if you’ve got very matte objects, then lighting doesn’t matter as much, especially if you’re using structured light, 3D cameras that project patterns. And then they’re looking for the projected light coming back. Then the lighting really doesn’t matter quite as much. If you’ve got shiny objects, though, you do want to avoid having too much directed light focused on the objects. They can bounce back in unpredictable ways and cause blind spots for cameras. That can cause problems for 2D or 3D cameras, as well as for humans sometimes. So you want to minimize that sort of thing just by diffusing the light, not necessarily blocking it out completely but diffuse lighting. And if you’ve got a mix of parts, it’s always safer to just use diffuse lighting where you can.
Jimmy Carroll: [00:20:49] We haven’t really touched too much on specific industries, so I know CapSen is involved in several — general manufacturing, automotive, logistics, and warehousing. But one that I wanted to ask about, because I know you and I’ve seen some of your videos and everything, is the medical space. Your technology’s been deployed here in some really interesting ways, especially over the last few years with things like COVID-19, obviously, and some of the applications that have arisen as a result. So maybe without naming some of the customers, can you talk in broad terms about some of these recent deployments?
Jared Glover: [00:21:25] Yes, so we’ve been getting a lot of interest from the medical industry on using these types of bin-picking robots. So we’ve done medical packaging for packaging COVID test kits. We’ve done sorting, sorting medical supplies for hospitals, order fulfillment of cartons of medical supplies from larger boxes on flow racks. So really there’s a wide range of applications that are useful in the medical industry and gaining some traction. And it’s really just the start, and this goes beyond medical, but there’s so many applications where you’ve got objects that are randomly configured. You’ve got you’ve got stuff randomly piled in bins or on shelves or inside of a box. Even if it’s somewhat structured. Maybe it’s aligned in grids, but you really don’t know exactly where the rows of the grid are and where the separators are. Or maybe they’re packed tightly together. In all those situations and any situation where you don’t know exactly where something is every time, you need vision, you need some intelligence to the pick and place with the robot. So there’s lots of applications in medical that we’re gaining traction in for sure. But it’s just a microcosm. The problems that they have in medical with bin picking are the same problems in every other industry. In medical they’re having such acute supply chain issues and labor shortages that are really affecting the supply chain. And like in medical, if you have issues with the supply chain, somebody’s surgery is getting delayed. So it’s really big consequences. Whereas if there’s a delay in the shipment of some nuts or bolts or something, maybe you get your Tesla a couple days later, but that’s not going to kill anyone. So medical has a very acute need for more automation. That’s where we’re seeing more traction there right now.
Jimmy Carroll: [00:23:34]. Supply chain. So you mentioned supply chain. This is a topic that I wanted to bring up too. So, Association for Advancing Automation, A3 — their figures show that robot sales hit a record high last year. North American companies ordered more than 44,000 robots, close to $2.4 billion. These numbers represent increases of I think 11% and 18%. So automation is obviously more important than ever before, and it will continue to be. On the topic of A3, at the business forum this year, during Alan Beaulieu’s Global Economic Outlook keynote, which anyone who’s been to the show knows is a great annual presentation to catch, he said automation will be key to keeping the economy going during uncertain times. So I say all this because this is all very exciting stuff, but what are your key takeaways from all of this? What are some ways that robot sales can continue to grow, including new or expanding industries?
Jared Glover: [00:24:30] So it’s interesting. The growth story in robotics in North America, particularly the U.S., recently there’s been a lot of growth, but I think most of the growth has been playing catch-up. Because countries like South Korea and Japan and even China recently have really surged ahead in robot adoption. They have way more robots installed per capita than we do, especially South Korea and Japan and some European countries. So we really lag behind here in the U.S. for the last 10 or 20 years. So we’re just playing catch-up right now. It’s long overdue and necessary. A lot of the growth has not required a ton of intelligence. It’s not required a ton of vision systems and AI. But the next big wave of growth after that obviously is going to be with AI-enabled robots. And that’s an area where the U.S. is number one. The U.S. is number one in AI software. And so we can really capitalize on that here and get an advantage in the types of automation we have here compared to other countries. I just read an article today that was talking about Japan having this huge lead in robotics — the hardware of robotics — for so long and adoption. But they’re lagging behind now in terms of AI and the software aspect. So I think that’s going to be the story in the coming one or two decades, is that the U.S. is going to be the center of robotics again because of our advanced AI capabilities. But we need some help from government too. We need them to give companies the incentives to increase adoption faster, because there’s still some things holding us back in that respect.
Jimmy Carroll: [00:26:21] Interesting. So staying on the topic of the general space, the general industrial automation space, what are you most excited about, both in the general space and for your company, this year and beyond?
Jared Glover: [00:26:35] I get excited every time I see a robot do something new. So whether it’s mobile manipulation, putting arms on mobile robots, I think that’s going to be a huge growth vector in the coming years. Whether it’s the fact that robots can handle more entangled parts or tend things that are moving or packing objects into really tight spaces and getting the motion control right for that. I get really, really excited about things like that. We just solved a repackaging problem for an automotive company recently. That’s something that has been a struggle for robots before. Shiny and transparent objects. So that’s what gets me excited. And I think all those things — just adding new capabilities to robots and going mobile with manipulation. All of those things are going to be big drivers of growth.
Jimmy Carroll: [00:27:30] Jared, you mentioned it earlier. Your technology will be in a couple of partner booths at Automate. How else can people see CapSen this year or otherwise? How can they get in touch with you if they have questions?
Jared Glover: [00:27:41] Yeah, you can check out our website, capsenrobotics.com, and definitely come visit us at our booths at Automate. We’ll be at the Denso booth and the Calvary booth at Automate in Detroit this May.
Jimmy Carroll: [00:27:54] Well, Jared, thanks so much for joining us today and sharing all these insights. I really appreciate it. It’s always a pleasure to talk to you.
Jared Glover: [00:28:00] Likewise. Thanks for having me.
Jimmy Carroll: [00:28:02] Of course. And everyone tuning in, thanks for watching. We look forward to sharing more of these conversations with you in the near future. If anyone has any questions or ideas or would like to join the podcast, please visit us at manufacturing-matters.com. And thanks very much, everyone, and have a great day.