A Smarter Way to Manage Supplier Performance in Public Procurement

Coming to you Live from NASPO's 2021 annual conference in Austin, TX! We talk with Dan May, Deputy Director of Operations, in the great State of Florida, about their Vendor Performance Tracking (VPT) system. Dan provides us with an overview of the improvements and evolution of vendor evaluation. We discuss how performance data provided by vendor rating systems can increase transparency and accountability throughout the contract life cycle.  Our speakers will share their experience implementing an evaluation process and system.

Click here to view a transcript of this episode.

Follow & subscribe to stay up-to-date on NASPO!
naspo.org | Pulse Blog | LinkedIn | Youtube | Facebook 

Transcript

Copy Transcript Copied

Kevin Minor: 0:01

This is the first Live Pulse podcast. You are a part of history, my friends. We got a really great show for you today, really happy you guys are here. I am Kevin Miner.

Josh Descoteaux: 0:19

I am Josh Dakota.

Kevin Minor: 0:22

Yep, that's right. Checks out. This is the podcast where we are monitoring issues in state procurement. We have our finger on the pulse. It's going a little fast, that's okay. So we're excited to be coming to you live from NASPO Annual in Austin, texas. Are you guys having a good time so far? And we are so excited to be joined by our very first live guest. This is Dan May, the Deputy Director of Operations for the great state of Florida.

Dan May: 0:52

How's everybody doing today?

Kevin Minor: 0:55

Dan, thank you so much for coming, sir.

Dan May: 0:57

Well, thanks for having me. I'm kind of excited to be here this first live recording.

Kevin Minor: 1:02

Dan, you are the Governments in Training. Deputy Director at the Division of State Purchasing in the great state of Florida. Can you describe your roles and responsibilities and why we should be listening to what you have to say today?

Dan May: 1:19

Honestly, you probably shouldn't.

Dan May: 1:22

Transparency folks so yeah, important in public procurement, right Transparency? No, I took an interesting path in that my background is actually in counseling, mental health and substance abuse work, and then somehow I got into the training world and that led into procurement, and now I have a multi-billion dollar e-procurement system underneath me, which makes no sense to me. But no, it's been a lot of learning drinking from the fire hose, if you will, but my responsibilities now I have the certification and training programs for the state of Florida, including our certified contract manager program, which relates a little bit to vendor performance. And then for us, we're so fortunate to have our full procure-to-pay suite as an e-procurement system, and that suite includes the vendor performance tracking that we're going to talk a little bit about today. So I might not know what I'm talking about, but I can at least pretend like I do.

Kevin Minor: 2:22

Absolutely, and you guys probably won't know the difference. So hopefully anyway. So what's your background? How did you kind of come into this position?

Dan May: 2:31

So I learned very quickly that, as passionate as I was about mental health, particularly working with adjudicated youth, it didn't pay the bills for a small family. It didn't pay the bills for a small family. So I started getting into training work. I did a lot of curriculum design, development, oversight, delivery of training. I actually got hired in state purchasing to be a training coordinator and they were like you know, come coordinate trainings for us. You don't have to supervise anybody and we'll pay you more than you're making now. And I thought where's the bad side of that? And then over time got promoted to governance and training manager and now promoted to the deputy director.

Kevin Minor: 3:14

So we got some pretty exciting stuff planned today. We got some games, we got some questions and answers and at the end we got audience participation. Okay, so that's you guys right, but first I think we need some context, dan. So we know Florida uses a custom application called Vendor Performance Tracking. It's a VPT system to evaluate vendor performance. Can you provide some background on this application?

Dan May: 3:41

Absolutely, so, like I said, we're fortunate that it's embedded into our larger e-procurement system and essentially what we do any transaction that's over $2,500 or that is a purchase off, a state term contract, an agency term contract, through an invitation to bid, a request for a proposal, an invitation just about every transaction we do the system sends an automatic survey to our agency customer and asks them to rate on a scale of one to five, with one being pretty much fails to do anything and five being the best vendor of all time the vendor's performance in four major domains, and those are performance according to specs, invoicing, delivery and customer service. So we're trying to keep it simple for our agency customers and yet, at the same time, really identify potential places where the entire delivery of goods and services could break down.

Kevin Minor: 4:43

Interesting. And what's the significance of one through five? Why not one through 10 or one through 20?

Dan May: 4:50

Yeah, we just didn't want to do one through 20.

Kevin Minor: 4:53

Transparency again, yeah transparency.

Dan May: 4:56

No, you could do, and if you're a state or agency that's beginning this journey, I think develop a rating system that makes sense for you. For us, we felt like one through 20 has so much variation and potential for confusion it was a little too much for our purposes. One through 10 really gives you that back in school grading scale you know A's, b's, c's, d's. We didn't really want that either. So we went with one through five because we felt like it made the most sense for us. It was also the least burdensome to our agency customers, because that's. Another consideration, obviously, is the amount of work you're asking people to take on through surveying, follow up. But also, around the same time we've been doing this for almost 10 years now. Around the same time, we had an initiative to improve employee performance evaluations through this one through five scale, and where that landed was anyone who was doing their job 100% of the time.

Dan May: 6:01

You received a three, which is the corresponding for our vendor. You deliver, you deliver on time, you deliver what you said you were going to. It's a three Slightly better than that. You get a four, exceptional five. You know a little worse two Right.

Kevin Minor: 6:18

That's interesting, though, because I mean, whenever I rate something, I'm either If I'm going to take the time and energy I'm either going to rate it really poorly or like exceptional. So it's kind of surprising to me that that three is kind of that average Right.

Dan May: 6:34

And there is the potential for bias, right, how many of you shop online where it's asked you to give stars and typically we give one star or five star because it's either the greatest thing we've ever seen or it was a terrible experience? So we have to be mindful of that potential bias that could show up in these results. We do a lot of outreach with our agency customers through my Florida marketplaces, training teams, reminding them of the purpose of this, the sort of what's in it for me, how it can really help you understand your vendor better, look for patterns of performance, those kinds of things. So there's a big educational component that goes with it, because the flip side of the everybody gets a three is the Oprah right. You get a five.

Kevin Minor: 7:17

You get a five, you get a five.

Dan May: 7:18

We don't want that either. We want authenticity, which is what we're looking for.

Josh Descoteaux: 7:24

Yeah, and Dan, I think this comes at a really good time too, because you mentioned shopping. We are all checking ratings all the time for what we buy, the Uber drivers that drive us around restaurants looking at the menus and just all the comments there. So, really incorporating this and bridging the gap between that and public procurement when you're talking about the four domains, the four categories that you all came up with, looking at that and having the agency customers really understand what they are rating, making it broad enough but specific enough for what they're buying. So how did you guys come up with those?

Dan May: 7:59

Right. So, again, something you're considering in the future. There's a wide array of categories you could consider. Right, we went performance to specs because felt like that was part of that contract compliance need. Right, did you actually follow the terms of the contract? And then delivery. Obviously it's a very clear expectation there, whether it's outlined in the contract, in your purchase order terms and conditions, wherever you might find it. There is a specific delivery date, delivery location. These were more black and white, a little bit less subjective for us. We also chose to look at invoicing. Felt like it was important in the overall vendor performance arena and contract oversight. Did they actually deliver what they billed you for? That matters site did they actually deliver what they billed you for? That matters. And then the last, because all of us in procurement are really in customer service, we felt like we should assess overall customer service ratings for the vendor. So those were the four.

Josh Descoteaux: 8:57

Has there been blowback? What's that been like? When there is a vendor that is rated low when they were expecting a better rating? They think that they're delivering what they should be. What's that situation like?

Dan May: 9:08

So for us because we just use this as one tool when we're considering vendors.

Dan May: 9:14

Right, this is not our, it's not a factor in past performance for procurements, it's not the beginning of some legal process for noncompliance.

Dan May: 9:24

It's really just a look at data and an understanding of customer satisfaction with our vendors. So we have had a little bit of frustration or a little bit of that. You know I can't believe I got a two, but never anything where it really pushed us into a legal arena or concerns from that. Because for us, unlike some states which I know is now statutorily mandated hands, if you're required to have vendor performance tracking, a couple of you, I see two. I think that makes it more of a challenge because then you're really locked into doing this and you have to consider it as past performance or as part of your contract managing process, whereas for us it's another data point to help and that's one of the other selling points for our agency customers is we try to let them know look, this is a chance to hold a vendor accountable but also set expectations, build your relationship, recognize them when they do really good work. I think sometimes that part gets overlooked, but it's another factor.

Kevin Minor: 10:35

So what can encourage agency raters from just clicking all the way down, right, just all threes, all the way down, so they can go home? What are the dos and the don'ts from the agency perspective?

Dan May: 10:49

It's difficult to manage and, I'll be honest, sometimes that's a challenge because you just want to go home, so let's just click through these surveys. But one of the things is, because it's an auto-push survey, we time limit the time they have to respond, but we try to give them more than 24 or 48 hours. We actually give them a full week to submit that survey and explain. We know that you're busy. We just want you to follow up with us. When you have time, when it's appropriate for you and when you have the energy to spend five minutes thinking through what this vendor really delivered for you, spend five minutes thinking through what this vendor really delivered for you.

Kevin Minor: 11:29

And so what's been the impact on the system as it relates to the process and the procedures right Like have your terms and conditions changed or the RFP and the contract?

Dan May: 12:09

I can't say directly because this is a tool for us, but I can say that there have been some instances where if you're looking at vendor action, the complaint to vendor so suddenly vendor performance tracking data is going downhill, Complaint to vendors coming from agencies are increasing. I can think of one specific instance where that combination kind of led us to some meetings with a vendor and we realized that they were in over their head, unable to fulfill the state term contract as it needed to be done, and we wound up terminating early and re-procuring that commodity Wow.

Josh Descoteaux: 12:40

And it was painful. That sounds painful and you had mentioned. In order for the response to get out there, you guys designed some automated prompts and notifications so that the agency customers would be able to see that when an invoice comes in, or explain or talk about the automation part of the VT Once the good or service is delivered and the agency clears the invoice in our e-procurement system, that's what triggers the survey automatically to be sent to that agency customer.

Dan May: 13:11

We don't allow them to delegate the surveys, which is something else that we felt was important. Like, I don't want to send a survey for goods that Cindy bought and have her delegate that to Jordan because he doesn't know. Necessarily, jordan doesn't know. Guys. That's your takeaway from the session, yeah.

Kevin Minor: 13:34

I think that's a song, isn't? It Could be. Jordan doesn't know Jordan doesn't know.

Dan May: 13:39

Yeah, okay, so you know. I have no idea what I was talking about, cause I distracted myself with my own bad jokes, dan doesn't know, Dan doesn't know.

Kevin Minor: 13:49

There you go.

Dan May: 13:50

Jordan, but using that sort of auto function without delegation. That's one way we have a pretty high return rate, which is one of the considerations you should make. We'll talk about that in a little while.

Kevin Minor: 14:04

Yeah, so do you find that that conversation with the vendors ever becomes condescending? How do you? How do you do that? Right, because there's no way that you can't have a discussion with a vendor, tell them that they're doing poorly and then lead them to the solution without being condescending, right?

Dan May: 14:22

So I don't have a single bit of sarcasm or condescension in my body. Oh really.

Kevin Minor: 14:29

None, not even able to do that, not even a little bit. Not even a little bit.

Dan May: 14:33

Yeah, but no, it can turn very confrontational, condescending, hurt feelings, frustrations. You guys have had conversations with vendors that weren't going to be positive Interactions. Right, how fun is it? It sounds like very. It's awesome. So this is the part where I stand up, because I have to do the movement and those kind of things. So we're going to key you up. It's a little after lunch. I know some of you sleepy eyes. I see them a little bit. I will throw candy at you. I have no issues with that. I've seen him do it. They saw it in the last group.

Dan May: 15:14

The other thing we're going to do is, every time I say the word magic. From this point forward, you're going to go magic.

Kevin Minor: 15:20

Give it a shot Ready One, two, three.

Dan May: 15:30

Here is a quick slide of the five one through five scale that we use to rate our vendors. You can see one there is a pretty bad experience. Two is you know maybe they did pretty well but the delivery was late. Maybe customer service didn't return a phone call. Three is you know meeting what we're absolutely looking for. Four gets to that you know you achieved a little bit better than what the contract said you needed to. Five is you just blew our minds with everything that you delivered for this particular order. Those are the four domains that we talk about, but this is the magic part.

Kevin Minor: 16:11

Yay, all right, you guys are with me. Very good, very good, give yourself a round of applause. Yay, that was beautiful.

Dan May: 16:17

All right. So this is an example rating and an example conversation you can have with your vendors. If I told you there's a way to help them understand where they're coming up short, see the things that they're doing well, and they can give you their own corrective action plan without you ever having to say anything negative to them, what would you say? Awesome, it's magic. Okay, so this is how this, in an ideal world, works. It doesn't always go this way, but it's a great starting point if you're using this as a part of your tools for vendor management, vendor relations. So this particular vendor, customer service we're going to see that they, in the past month, had an overall rating of 3.7 on 18 invoices. What does that mean? That's the question I want the vendor to tell me. You guys, you see this vendor, you see this 3.7,. You're having this conversation with the vendor, you meet with them and you say, okay, 3.7 this month in customer service. What does that mean to you and what is the vendor telling you? Hmm, what are they saying? They're better than the 3.7. They're going to stay at the top saying, okay, they might say there are four or five, but if I'm actually asking you, explain to me when you see 3.7, what does that mean to you as the vendor? Remember, we're also talking to them about the fact that our ideal rating, or if they met their objective 100% of the time, was a 3. So a 3.7 is way above average. I'm exceeding Above average. I'm actually doing better. They might not see it that way. They might, they might not. It could say room for improvement. They could say absolutely room for improvement, absolutely. But they're also going to have to tell you what does that 3.7 mean to you? It means what they have interacted with and what they have done with your customers, right? So it's a starting point. So the second part of the conversation, and this one to me, is the real, the key question, because this is the part they love. This is where you're really going to build some rapport, identify some positivity.

Dan May: 18:39

Okay, so, for this month in customer service, you got a 3.7. It wasn't a one. So tell me what you did. And it's important that you go 3.7, not one, because you want them to be able to talk about some positives. Is it important that we have our vendors talk to us about the things they are doing? Well, absolutely. Number one, because it helps them understand you're not always the bad guy. Number two, because it reinforces the behavior you're actually looking for. And number three, because it helps build relationships for the future. And if we're really trying to strategically source, especially in certain services and commodities, then we need to build these strong relationships.

Dan May: 19:27

So, vendor, you're at a 3.7 and not a 1. What did you do to earn that and what are they having to tell you? And not a one? What did you do to earn that and what are they having to tell you? All the positives, all the hard work. And then this is the question where it gets a little more challenging. Okay, you're at a 3.7 this month. What would be different, what would change for you and for our agency customers if you were able to move from a 3.7 to a 4? So what would you do differently? What do you think the customers are looking for differently to help you move? Just that little bit. I'm not looking for a 5, right, that's unrealistic. I'm looking for a four, just a slight improvement. And now, what are they telling you?

Kevin Minor: 20:22

What they can do to improve.

Dan May: 20:24

They're giving you an improvement plan just by you asking the question. So you're not demanding anything, you're not telling them they're doing anything wrong. You're just thinking about how do we constantly improve our customer service. It's not necessarily a contractual issue. This is a relationship building tool now and it's a very small change we're looking for. So maybe that improvement would be hey, we'll do a secondary quick review to make sure all of our invoices are accurate. Maybe it was nine out of or maybe it was 16 out of 18. This month We'll shoot for 17 out of 18. Small improvements.

Kevin Minor: 21:06

Continual small improvements. I feel like this would work really well with toddlers too yes, toddlers.

Dan May: 21:13

It does. It also sometimes works on spouses, oh rating spouses.

Kevin Minor: 21:18

That is another general session. That's a whole different thing. It's a different conference. Don't really do that.

Dan May: 21:24

We're not going to, it doesn't end well.

Kevin Minor: 21:26

Yeah, disclaimer the NASPA Pulse does not support that.

Dan May: 21:30

But I will give you this. You guys heard about my background in mental health and substance abuse. This is actually called the readiness ruler or the motivation to change ruler, and these three questions come from 1980 substance abuse literature and it's been proven over 40 years. This simple chart actually changes people's behavior and it works for vendors and it works for anyone needing change. It's a great way to talk to your employees about their evaluations. Hey, you're at a 3.6 this year. What would be different if you went to a 3.8 for next year? And guess what? They're giving you their own corrective action plans or improvement plans, Right?

Kevin Minor: 22:11

Leading them to the solution.

Dan May: 22:12

You're leading them to the solution in a kind of non-directive way.

Kevin Minor: 22:15

Non-condescending way yeah.

Dan May: 22:18

Yeah, but depending on the person, sure.

Josh Descoteaux: 22:21

And so the 1.5, as I saw it, you know, like one star, two star through five stars. This this really gets down to, it's not just numbers, it's the relationships, right, it's how you're talking to the vendors and it is providing a way, a vehicle to get to those conversations to be able to say that.

Dan May: 22:40

That's a great way to explain that it's not about the numbers. In reality, it's about the vendor relationship, it's about vendor accountability. Now, if they're down here at a 1.3, we're having a different conversation, right. 1.3, we're having a different conversation, right, you can still use it a little bit, but I don't really need you to go from a 1.3 to a 1.5, because that's still not getting us what we need to our agency customers. But for you know, if you're kind of in this range, this is an effective tool to try to get better.

Josh Descoteaux: 23:15

Dan, that is awesome. As we wrap up here in this range, this is an effective tool to try to get better. Dan, that is awesome. As we wrap up here in this session of the podcast, do you have any advice for our listeners and our audience?

Dan May: 23:26

Maybe a few things. I know that for some of you, your states are experiencing legislative mandates to add vendor performance tracking. Maybe it's something that you're just pushing internally as an initiative. There are a few things to consider. You know we're very fortunate in Florida to have our custom-built system be able to add this. I'll also go and say I think attached to your breakout session information is a link to our webpage. We are the public records law state, so all of our vendor performance rating information is online. Any of you can view it if you want to.

Dan May: 24:07

But the few things I would say consider. Number one is response rate. You want to either procure or build a system that people will respond to, because otherwise you're just going to have nothing. If you're having 10,000 invoices go through a month and you're getting surveys on 70 of them, is there any validity to that? Sure, probably not. So think about your response rates. The second thing I would suggest is find the rating and the metrics that work for your state, or find the rating and metrics that are based on what you need.

Dan May: 24:48

Maybe these four domains don't work for you. Maybe you need six. Maybe you do need a scale of one to ten. Maybe you want to use the star system. Just think about what you really are looking for this to be. And that's the third thing is, if you're collecting the data, if you're going to ask your state agency customers, your eligible user customers, to go through these extra steps of responding to surveys, rating vendors, what are you doing with it? Is it just to check a box? Because, if so, don't waste your time, don't waste everyone's time, but is there a strategic purpose for it? Are you going to include it in past performance? Are you going to use it as a vendor relationship tool? Are you going to think about it in some other terms that I haven't thought of yet and don't have the words for right now?

Kevin Minor: 25:38

Right, Yep, and if you have the words, we can fix that in post later. Okay, cool.

Dan May: 25:43

Yeah, yeah, we'll just edit those in right.

Kevin Minor: 25:45

All right, awesome Dan. Thank you so much for talking with us today. I really appreciate it. Can we get a round of applause for Dan? Thank you guys, absolutely. Thank you, okay. Thank you Bye.