FreshEBT

Transcript

If you're thinking about Snapchat or Snap Inc., you're in the wrong room. If you're thinking about the Supplemental Nutrition Assistance Program, very good. You're getting prepared. For the rest of you, SNAP is also known as food stamps. It is one of the most important safety net programs in the United States. It's the biggest domestic hunger safety net program. It feeds over 40 million Americans each month. It's available in all 50 states. It's also available in the Virgin Islands, DC, Guam, and Puerto Rico. So the way it works is, if you apply for SNAP and you fill out a form, you can either mail it in, fax it in, or walk to your county office and give it to them. Thereafter, typically, a county official will schedule a phone call with you where they'll interview you to determine your eligibility. At this point, if you qualify, they'll also determine your monthly benefit amount and you'll start receiving food stamps every month.

In the past, this meant that you would actually receive paper food stamps. In the modern era, when we're used to just being able to spend plastic money, this isn't particularly convenient. It also sticks out as a sore thumb that you are pulling out this other kind of paper currency. However, today, the way it's done is through this thing called an EBT card. It's pretty similar to a debit card, except you can only use it at stores that accept EBT as a form of payment and you can only use it to buy eligible items. So once you qualify, you receive this card in the mail and then you can use it to buy things at stores.

This was a huge achievement, by the way, because this moved off of these paper ticket systems, which are still used for other benefits but not for SNAP anymore. And just so you know, this EBT system is used for other benefits as well. It's not just food stamps that are electronically distributed on this card. There's other benefits. One of them is called TANF, Temporary Assistance for Needy Families, which is directly cash. Another one is called WIC, and a large F number of the births in the United States are actually supported by WIC. It stands for Women, Infants and Children. And these benefits are also sometimes loaded on these EBT cards.

I am going to introduce myself now. I'm Ram Mehta, the CTO of Propel. Propel is a small startup based out of Brooklyn, and we build software for low-income Americans that are often overlooked by traditional technology innovation. Over the past few years, we've built and scaled an app for food stamp recipients called Fresh EBT.

Today, Fresh EBT focuses on three things. It helps people manage their benefits. That's the core. That's where we started. It allows people to save money. And we do this by partnering with lots of different types of organizations, social service organizations, government organizations, and we also vet thoroughly fintech products that we'd like to propose to our users. For instance, earning is a big one. And the third thing, this is something that is relatively new, is we are trying to focus on allowing people to earn income and provide an opportunity to do so. Sixty percent of Fresh EBT users are underemployed, and we've made it a goal to help connect users to new job opportunities and other ways to earn income.

Now, not all of the people who use the app can work. We serve a sizable number of people who are long-term disabled. But even people in this situation have expressed excitement about other ways to earn small amounts of money. So Fresh EBT today serves over a million people each month and they use the app quite often. They use it for a variety of things too. And so what I'm going to try to cover in this talk is the journey of taking this app with a very, very small team, from one state to all 50 states, and from zero users to over a million monthly actives in the course of two years. And I'm going to be jumping back and forth between lots of different topics. Some of them will be quite technical and some of them will be product related. And scaling in the sense is quite ugly because a program as complicated as SNAP, it has so many stakeholders and it has so many edge cases that it's not just the technology that you scale, it's that you have to scale all of this complexity at the same time.

Fresh EBT began with a really simple problem: EBT cardholders had no easy way to check their balance. What did they do to check their balance? The most common way is to call this 1-800 number. You then dial in your 16 to 19-digit card number every time you want your balance, and then you wait for a robot voice to tell you your balance. Another common way is to actually save receipts, because when you make a purchase, your last receipt has your balance amount on it. This can get really cumbersome because if you misplace the latest receipt or you mistake your latest receipt for another one, then your balance is off and you have a really embarrassing experience at checkout where you do not have enough money to pay for your groceries. So there exists a website in most states, but when we interviewed people, basically no one talked about it. This website is not mobile friendly. It's not easy to use, but it does exist.

Early Technical Decisions

I'm going to pivot a little bit to the technology now. When we started out, we had a very small team. We had only three people on the team and only one engineer. I was the one engineer. And we had to make a bunch of decisions at the outset. And so, I'm going to come back to some of these things as we go through the talk, but I just wanted to set some technical foundation for where we started. There was a single developer at the beginning, so the world was moving towards microservices, but as a single developer using a monolithic codebase is really appealing. However, scaling a monolithic codebase as you get more usage is not appealing.

Even at the outset, splitting it up into two simple services, one that did all of the scraping, and another service that did everything else, went a long way as time went on because it was easy to scale these two components independently. For those of you who are familiar with Supervisord, I'm a big fan of it, I think of it as just like a poor man's Docker, it gives you some level of all of the things that Docker gives you. It's just not as much isolation, but if you set up your system like this to then containerize it, it’s quite straightforward.

This is something that's a little - I believe in this a lot, in continuous integration and deployment at day one, and it helped that this project actually had teeth to it and took off. But in my view, if you have a complicated enough project and you're going to be making a lot of releases, doing this early even with a single developer went a long way. It meant that when I did releases, I wasn't actually looking at my terminal waiting for it to finish. If I was on a flight and I did a release, I didn't have to worry about my WiFi leaving me in the middle of my release. So even from day one doing this.

Day 1

There were so many consequences of having continuous integration and deployment at day one that I didn't even realize until we were farther along. This is kind of a recap of what I just told you. When we started out, we built an Android app. We had the servers that I described to you, and we'd built one state, the state of Pennsylvania. And at this point, I knew nothing about the rest of the program or how it worked. And we built this one state by allowing people to register through the app and we'd scrape their balance from this website.

I didn't even know who operated the website at the time. So the next step was we released this to the App Store. It was right around Christmas time. And then we saw people using it. It was really exciting to see that people were using it. Then the next question was, “Okay, we should make this for as many states as we can.” So then I started learning more about the SNAP program. And this is very inaccurate but, my understanding of how it works. There’s so much more that goes on into it, but if you notice that the online portal is just a small part of this whole system.

The way it works is, the money for the SNAP program comes from the Farm Bill. It's a piece of legislation that gets renewed every five years. And even though the money comes from the federal government, the program is actually implemented individually by each state. They decide how to do it, and they typically hire a private contractor to implement the EBT system. This private contractor has to figure out how to get everyone their EBT cards. They have to figure out how to get all the retailers to be able to accept EBT that want to. They have to do POS integration, these point of sale machines, to know which items are actually eligible for SNAP. And then amongst other things, they have to provide an IVR line, which is the 1-800 number that people call, and they have to provide this online portal.

All of this is negotiated in the state EBT contract, and the online portal is a small, small part of this whole thing. What this means is, this kind of complexity in the program at an administrative level, it has direct implications on the technology that underpins these systems, not just the online portal, the actual electronic benefit transfer system itself. I have dozens of examples, but I'm going to try to keep them to just a few that I can use to make some other points.

Here's one of my favorite ones actually. So in some states when you register for this online system, you have to choose a security question and answer it, which is pretty common for a lot of products. And then if you want to reset your password, you'll be prompted for a security question again. The only problem is when you're prompted during the password reset flow, it doesn't tell you which question you've chosen. It just gives you a drop-down with all eight questions. And if you happen to not choose the right question, the error message is identical to as if you answered it incorrectly. So you had better remember which question you answered, otherwise good luck.

This happens because no one was looking out for the user experience. They're trying to make sure that an online portal exists and it meets all kinds of criterion, each page is printable or something like that. But things like this get slipped through the cracks. And then speaking of registration flows itself, each state has a totally different kind of registration flow. The two on the left are actually done by the same EBT contractors, so they're kind of similar. But even amongst those, you'll notice that the fields that they ask for are quite different. The password validation requirements are all over the map. Some of them are kind of senseless. There's one state which had a password of six to eight characters, which doesn't seem particularly secure to me.

When we started out, it was still three people, one engineer, and we had to take on 50 of these, or a large number of these kind of forks. So for me, philosophically what this meant was, I had to be able to use my colleagues' brains to help me even though they didn't want to write code. And so this is what I sort of stumbled upon.

Empower Those Who Do Not Wish to Write Code

Writing JSON, or YAML or whatever it is, is a lot easier than writing actual code. There's no control flow. It's not a real programming language, it's just a description. It's configuration. It's the same reason as programmers why we like infrastructure as config so much. Features as config works really well for non-programmers. So, essentially, everything we'd made, we'd make it highly configurable and we'd make it so that my non-technical or my non-programmer colleagues could maintain features themselves and do customer support for these features. That's not to say that I wasn't a part of it, but this kind of removed me from the immediate loop.

So what does this do? This particular JSON is just a config for what an inapt flow for registering in California would look like. I'm going to show you this particular JSON now, not written by me, written by my nontechnical coworker, spits out like a edge flow that looks something like this. By building features like this, we were able to scale this kind of stuff up quickly with a really small team. We were still just three people. The screenshots are from the modern version of the app, so it didn't look like this back then. We didn't have a designer on the team yet, but it effectively worked the same way.

Day 35

At this point, we are able to scale through a large number of states. We still had a long way to go. We had a few users and we started getting validation that this is something that's important and is solving an important problem. And now it is time to build it out for the rest of them. Now, unfortunately, not every state actually has an online portal. Most notably amongst these is Texas. Texas is a big state, it's a big population. We felt quite bad that we couldn't serve them with this product. So just for Texas and Massachusetts at the time, we had to build something that... So the only way you can get your balance in these states is to actually call the 1-800 number. So this was our only option as well. So we would do that, we'd record it, send it to Google's transcription service and then send it back to the user.

This is a substantial lift, right? It goes from being a single web request to scrape a page, to going to something where you're pulling a phone call to wait until it's finished transcribing and then giving the result back to the user. So this meant that the front end also had to change just for this flow and starts looking something like this. You put in your card number, you see some loading bar that makes up messages when the phone call is happening in the background. The funny thing is, those made up messages would make me believe that it was actually doing something too, because after sitting there for hours … and they still do. When I'm testing, I have to remind myself, "I wrote those. They don't do anything." But yes.

Day 90

So 90 days in, with the Texas hacks, if you will, we managed to get coverage to all 50 states, and this was a big deal. I say 52 there, because at this point we also supported the Virgin Islands and Guam. No Puerto Rico yet, but we'll get there soon enough. There's a particular screenshot I wanted to include that I forgot which had to do with Puerto Rico, which I'm going to just tell you about right now. When you open the Puerto Rico SNAP portal, the first thing you're greeted with is an alert that says, "If you're not using Internet Explorer between 6:00 and 9:00, please leave, the window will close itself now." It just does. It's also really fragile, their portal. So the moment we actually built a flow for it, we brought their system down in a day. So we had to scale it down a little bit and find other ways.

What we would do is if we were breaking the portal through scraping, they'd get really upset. But if we just diverted traffic to them, they didn't have as much of a case to be upset. So we'd allow people to register for the Puerto Rico portal from within the app, and all we'd do is when they click on the link, we would change the user agent to be IE9. So it would stop complaining and it would allow them to use it. Because the website worked okay, it just didn't want you to use it for some reason. Another thing I want to talk about is, I mentioned this earlier, that scaling is a big part of this. I talked about how we scaled to 50 states, and I that the capacity is also going to be something that we'd have to take on as we went from 7,000 to more users, and typically, conventionalism is don't solve problems that you don't have yet.

But my view is that when it comes to scaling, you want to have an answer for what is your most likely next scaling step. So you don't have to do it now. It just means that if you know that you can just double the size of the box and it will survive, that's a viable answer. If it means that you have to split out a service into another one down the line, that's a viable answer. But not having an answer to that question is quite frankly scary and leads to situations where you have capacity breaking your entire system and no answer for how to scale it.

Serving Low-Income Consumers Requires Understanding of Specific Needs and Constraints

Now that we'd built it for all 50 states and it was working, what was next? So we started interviewing our users and we started trying to get a sense of what was important to them. And one of the things that stood out here is, we were building products not for ourselves. None of us three had been on food stamps at this point. We didn't quite know what our users needed. So when we spoke to them, these are the things that came out, and these things translated directly into features in the app. When we learned that people share a smartphone with other people, we allowed them to add a second account. Now, this is atypical for a fintech product. You might be able to add multiple cards for yourself, but for you to manage your own account and your roommate's account, or your mother's account, if she's on food stamps, is not as common. So this, it wouldn't have occurred to us, I don't think, unless we talked to our users.

This is a really important one. So when I think of my banking app and I open it, what I care about is having the most accurate information present there, right? But food stamps recipients, they may or may not have enough data to be able to check their balance outside of their home, and they typically need to check their balance in stores before they actually make a purchase. So the offline experience of this app was really, really important; being able to cache the balance in the app, telling them when the balance was last updated. These simple things made it so that you could use the app effectively during your shopping trips, and prioritizing the offline version of the app as a result was very, very important.

The last is a grocery list. We've all made basic to-do apps as our hello world projects for any programming language. So the list is by no means amazing, but it has one feature that differentiates it from everything, and that's at the bottom; that subtraction where it totals all your items and it subtracts it from your current SNAP balance. That one subtraction makes all the difference. It tells you whether you're going to go over budget for the shopping trip. This, by the way, is really important to SNAP recipients because if you go over at the counter, you have this embarrassing experience where you don't have enough money to pay for the groceries and you're going to hold up the line as a result.

Month 9: Preparing to Scale

At this point with our new features and our fully scaled system, we started to see a lot of users. Most of our users came through word of mouth, growing organically. But also, because inherent to the company is providing value to those who are often overlooked, we thought translating the app to Spanish was really important. We added a designer to the team and we made an iOS version of the app. We translated the app to Spanish. And this is where continuous integration came in handy in a huge way. Things like copy changes and design changes could be done directly by the designer in Github, because GitHub now has a UI where you can actually make pull requests without even knowing how to use Git. And so she would translate all the strings and she just pushed the commit, and this would result in a new build, which we used in Dropbox. But she herself could then test to see if everything was working, removing engineering completely from the loop.

This happened in other places as well. It happened for server-side error messages, it happened for in-app error messages, you name it. Continuous integration made it so that we did something like 250 releases in the first year easily, and then another 200 in the second year. I probably would have done half as many if I had to do them myself, if at all.

Infrastructure vs Product

With such a small team, one of the big trade-offs is when do you work on features and when do you work on infrastructure. And if you don't work on infrastructure enough, then things break, then you get too much capacity. And if you work on infrastructure all the time, then you have no one working on features and actually pushing the ball forward. And this is where the modern era with Ansible and Terraform and these kind of tools, it's unbelievable. You can basically spend a few days on infrastructure and have something that takes care of itself and maybe sets itself for months. And presumably, if you use it correctly with other tools that cloud providers are offering today, you can set up scaling and all these things really easily. You pair this with things like Docker Compose and you get an amazing development environment.. So even with a really small team, using all these tools effectively was extremely important. In fact, it was the reason we could keep the team small throughout ,and still have a ratio of programmers to non-programmers which was quite small.

This talks about how you scale your infrastructure. You use these modern tools. It can babysit itself. How do you scale the features? I talked about this earlier. You make them highly configurable and you empower your non-technical team members to be able to figure out how to scale them themselves.

Year 2: Monolith + Microservices

So this is kind of where we landed a few months later when we managed to scale the app up a lot more substantially. Even with such a small team, we continued to use the same kind of ideas that larger teams would use. Even with the two-person team, we would have planning meetings, daily stand-ups, retrospectives. At this point, we split out our services. We still have a monolithic codebase but we actually have different services because we have enough services and they have different scaling properties that it does matter.

I'm just going to leave this slide up for a second because this slide is really important to me, because without attention to detail in sort of each of these categories, I don't think we would have made it. If he didn't care about the development environment, then it would have broken. So even though the team stayed small, we had interns on the team. We occasionally had other people come and contribute, and having a development environment that was easy to set up was extremely important.

There's another thing I want to touch on here. In this continuous integration point, when we first started we had the bare minimum continuous integration. So, basically, whatever script I would run from my laptop, would instead be run by Jenkins in the cloud. But then as we scaled features, moving to a model where we ... So, Jenkins has a feature. A lot of these CI tools have these features, but the ability to do continuous integration on each branch or pull request is invaluable. What that meant is, if someone was working on a feature and this produced a build that our designer and our product people could look at, they could check acceptance of this entire build without ... tests have gone on this build. They can accept it before merging it in, which actually in our workflow meant that the pull request at the end are often merged not by technical people, because they've been reviewed by non-technical people. I highly recommend using the multi-branch pipelines in Jenkins for this.

Scaling with a Small Team

Finally, I'm just going to recap the three key insights that I think allowed us to get to the scale. One is empower team members that do not code. I truly believe this, and I think that most people can be taught to maintain JSON or YAML configs quite easily. Invest in continuous integration and deployment early. This is kind of contrary to conventional wisdom where they say don't solve problems that you don't have, but I think that if you're going to build a lot of features, you're going to be pushing very often. So reduce your pain of doing this as early as you possibly can. And the third is, prepare for your next scaling step. Always know what you're going to do next.

In closing, I would like to say that we set out to help low-income Americans improve their financial health. Software companies often like to build products that they themselves would use. And as a result, the tech revolution has disproportionately served and benefited the affluent. I want to encourage people to look in non-traditional places to find problems and identify solutions that might not be immediately apparent or well known to them. These can represent big business opportunities, unique and complex technical problems and technological white spaces where a small team or even an individual can have an opportunity to make an impact. Thank you.

This is the team. The two dogs in the corner are our official mascots, so.

Questions & Answers

Participant 1: The app is free, right? So how does your business stay? Are you a nonprofit or are you a for-profit, and how do you stay in business?

Mehta: That's a really, really good question. We are a for-profit, and that has come with a lot of interesting and nuanced decisions that we've had to make throughout the process. The reason we are a for-profit was part of the hypothesis when we began was, can we use the tools of the for-profit world to attack a problem like this? Can we use the fundraising mechanisms of the for-profit world and the talent that the for-profit world can attract and still keep it such that we are able to build a sustainable business, but not at the expense of our users?

And in all honesty, I truly believe that up to this point that we have succeeded, but it's still an experiment and it's like, up to what scale can this succeed and what happens as you grow. And I think one of the most important parts of it is culture. You have to build a team where the values are such that you will always prioritize your users' needs. From a business perspective, it becomes really interesting because you have to find opportunities to not make money directly from your users. You have to find opportunities.

An example is we have a content business, we have a coupons business, and a jobs to business as well. So job postings are easy. They're a win, win, win. They're a win for us, they're a win for the employer, and they're a win for the person who applied for the job. Coupons are also interesting in that sense. Some of the things we value more are actual referrals that we do. People in this world are mired by scams and like these products, predatory products that are trying to get to them. And if you can vet these good products and separate them and actually put them in front of people, that's a huge value.

I think the reason we've been able to do that is inherently we've built a brand of trust. I think a lot of that has to do with the design and messaging and warm messaging, the kind of stuff that Isaac brought up in his talk. We know this from serving our users. We serve our users on all kinds of things often, and they tend to trust our recommendations because of that. So I think I've answered your question, but it's on many fronts that we try to make a business out of it.

Participant 2: This sounds very much like a perfect world. Everything's so nice and good. Can you talk a little bit about your uptime and what affects it?

Mehta: I planned to touch on that but I didn't. So the defining uptime is a little complicated because there's our uptime and there's the uptime of the systems that we rely on, and those systems are very flaky. When they start flaking, there's also, what do you do at that point? Do you try to escalate with the state? Do you try to escalate with the EBT contractor? Do you try to find a way to smooth out the load that you're putting on their system? And the truth is, we've tried to do all of these things in different situations. So there's our uptime, which is our own system, which we can be very thorough about and monitor really well. And we do this because it's easy for us to monitor our own systems. And then we also monitor all the state systems and then we have automated messages that show up, if necessary, telling people that your state's website is likely down. So a lot of it is being proactive actually.

Another common use case is the EBT contracts that states have with the contractors, they expire every few years. And when this happens, all kinds of stuff can happen. The website might change to a totally new website. The 50 states, this has happened many times, and they typically happen on weekends because they want to ease the burden on the actual SNAP recipients. There are many Sundays where I'll know that a contractor is switching and then we'll be prepared with as much of the information as we can beforehand. We'll message to our users that there will probably be a service outage at this time. So the truth is, we do our best to stay on top of it and we measure uptime, both of our system and the systems we rely on. And our uptime is a lot better than theirs, unfortunately.

Participant 3: Do you have an open line of communication with the contractors that build the sites, or what's that relationship like?

Mehta: That relationship is particularly nuanced as well, because when we started, no one cared. We were small enough that there was no impact. But as we grew, we became an important stakeholder in this ecosystem. And there are several contractors, and we have a variety of relationships with them and the state government themselves. We also have a relationship with the USDA, at different levels. And how this spans out depends specifically on the contractor and the state. And yes, some of our relationships are much deeper than others.

Participant 4: I was really humbled by your use cases; made me realize what a bubble I'm living in. And I wondered how you identified the need for this particular service?

Mehta: When we started we had a different product. It was called "Easy Food Stamps." It was a website that was designed to help you apply for food stamps. And as we were trying to promote this product, my two colleagues, they were interviewing users in stores and they saw someone actually call the 1-800 number, and that's when they got the idea that this is something that should be made a lot easier. Because immediately when they saw her call the number, they asked her a lot of questions. And one of the things that really stood out to me is they asked her the simple question, like, would you use a mobile app that did this? And she was like, "No, I'm fine calling the phone line, this works for me," because it's so ingrained at this point, they've been doing it for years.

But as someone who is using modern financial service tools, it was pretty apparent that this can be made more efficient. And hopefully if it's made more efficient, then people would use it. So I think that's the one point at which listening for problems is different from- you can listen to problems from your users, but they may not necessarily give you the solutions to them. I think knowing where the line is, is really important.

Participant 5: You mentioned with the food stamp program that only certain items are eligible for purchase. So I was wondering how your technology handles that, and is it a state-by-state basis thing? And then how does your application empower your users to know what kind of things are available?

Mehta: It doesn't directly affect us, it affects us in terms of customer service. In fact, we're a small startup, so we've been refining our values. Early on, one of our values was Google the pumpkins, because someone once emailed us asking us if they could buy pumpkins with SNAP. So we googled, and it turns out you can because they're a food item. As long as you don't use them only for decorations, as long as you don't say that you're using them for decoration, you can buy pumpkins with SNAP. It doesn't affect us directly because we don't do any of the retailer integrations. So this is more sort of the private contractors have to figure out what actual - what are they called? The barcodes or SKUs. What actual SKUs qualify for SNAP.

This problem is actually way, way, way worse for WIC though. WIC is the Women, Infants and Children benefit. The way it works there is instead of getting some money that's allotted to you every month, you're given like two quarts of milk, one bag of muffins or something like that. There's a very specific kind of category of things you get, or one 16-ounce can of beans or one 16-ounce can of peanut butter, and only certain cans will qualify. When you go to the store to buy this, it's like this insane experience for not only do you have to go through this list of stuff, but you have to find matching items and only those items can you buy.

What this also means is the stores have to be really well-trained on what items are eligible for different parts of the program. In states where they want to make this better, they'll actually try to release the list of SKUs so you can make a barcode scanner out of it. And there is an app that does this really well. It's called WIC Shopper. In other states, so to put this in comparison, New York still uses paper vouchers for WIC. So, it really depends on the state and a lot of other things. It's not like New York isn't progressive. It's that it's a lot more work to move New York off the paper system than it was for a smaller state.

Participant 6: You mentioned the difficulties with availability of smartphones for some of your users. And I wonder if you ever considered any non-smartphone option, like maybe a purely text-based solution, something like that?

Mehta: So firstly, I'm not sure I mentioned, most smartphones prevalence is pretty dominant in the U.S. I think 70-plus percent of the country is on smartphones right now. There are services where you can do a text-based one, and Code for America actually built a text-based SNAP balance checking system. In fact, the person who was involved with that now works at Propel and I had inherited some of their code when I built the Texas scraping system. It does exist and its viability is different in different states. But yes, you're right, that filling that need for them is important. But fortunately, today you can buy a smartphone for $20 at best buy, an android, which I think goes a long way.

Moderator: I think we're talking about the same person because that's how I found Fresh EBT, because I saw a talk years ago.

Mehta: That's right.

Participant 7: Can you talk a little bit about how you, in the early days, prioritized different features and that feature worked? Because I know that can be very tricky. How did your team manage that? What was the process by which you determined priorities?

Mehta: Really early on it was just about balance checking and stability, getting accurate information out in front of people. As we got to a stable point, then it became a little trickier. And then we tried to come up with a rubric for how we would do it. Our rubric was basically stability of the platform and ability to deliver balanced information was the number one thing. The second most important thing we thought was being able to put important content in front of people. So in addition to the content that we monetize, we put a lot of content out that we don't make money off of. So like, voting reminders, or tips on how to better use your benefits, and things like that. We prioritized things like that.

Then we tried to prioritize business contracts. So if there was an external contract that we had an SLA for, that would be the next thing on the list. I think there was a fourth category that I can't quite remember because it's been a while since we used that rubric. But basically we tried to match our issues against this rubric, because even with a really small team, the number of features and the number of fronts we were fighting on had gotten large enough that we couldn't keep track of it in our heads anymore. So effectively, we started using all the tools that bigger teams are forced to use and we were forced to do real prioritization exercises even with like three people in the room.

Participant 8: I'm curious - we talked about the contractors to implement these web portals. Have you guys looked at all at becoming part of those contracts in providing web portals, without having to scrape other people's things?

Mehta: We have. We're still a 10-person company that's really small, and these contracts are quite large. And again, the reason the portals are so terrible in part is because they are just an afterthought. And so if we could figure out a way to actually just own that piece, that would be really nice. But I think, at least at this point in the company's life, the idea of owning the entire process is something that we would have to take very seriously and consider very carefully before we moved in that direction. That said, it is something that's come up multiple times because of our proximity to that program. It's also complicated because if the contractors do a good job of making the system, that's not what our interest is. Our interest is in trying to add value on top of this process. We don't see ourselves as competitors to them, because it is a complicated program to actually implement. Yes. I hope that answers your question.

See more presentations with transcripts