Lauren Hill: We're very, very happy that Denise Juliano-Bult has agreed to give this talk. Denise is in our division, the Division of Services and Intervention Research, and she's in the services branch of our division, and she has a collection of portfolios that she oversees. And I think -- I said it right -- Denise say -- which portfolios she handles at this point, but she has a lot of experience and we're very grateful that she's agreed to join us today. So, Denise, maybe you can talk about your -- Denise Juliano-Bult: Okay. Thank you, Lauren. I'm very happy to hear your voice too by the way. So my name the Denise Juliano-Bult and I'm a program officer, as Lauren said, in the services research branch. It's actually Services Research and Clinical Epidemiology branch at NIMH, and I oversee two portfolios here. One is the Systems Research Program, so looking at service delivery systems and coordination across systems and delivering mental health interventions and services to people with mental illness. And the other is Disparities in Health Services Research Portfolio and that is what it sounds like, looking at -- studies that look at differences in disparities that are experienced in obtaining services by people with mental illness. And go to the next slide. I just wanted to kind of give people an overview of what the talk is going to cover today. I gathered from -- I got some background information from Lauren about this program and how people got involved and some background about who is involved. And so my talk is going to go from broad to specific. So I'm going to make some general statements about partnerships in research, what kinds of partnerships you could have or might want to have, things to consider about kinds of partnerships and all of that framed in the context of thinking that everyone here who's participating is thinking about applying for some kind of funding support for somewhere. And so they'll be those general statements and then I'm going to make some comments about applying for funding for NIMH and a little bit about applying for funding from other NIH institutes, what little bit that I know about what other institutes might be interested in supporting in terms of partnered research. And just outlining sort of help people sort of layout things that you need to think about when you're thinking about proposing this kind of research project. So the first question, obviously, would be if you're thinking about doing some research, who would be your -- who would be suitable entities to be your research partners? And it might be kind of obvious to say, but I'm going to say it anyway, it depends on the research question that you're asking, in fact. It really, really -- one of the primary things it depends on is the research question that you're asking, at least from the perspective from applying to an NIH institute. A place where, you know, there can be challenges or confusions or, you know, you may have already a relationship with some kind of entity that you really have a working relationship with and you feel they would be good for you to partner with, but depending on that what entity is, they may -- it will dictate what kind of research questions you could ask. So depending on the question you're thinking of pursuing, it might be clinics either mental health clinics or primary care clinics, it may be particular kinds of providers if you're -- what you're thinking about is an intervention strategy that is really intended to be provided by a particular kind of provider. Could be consumers, obviously, could be schools or states, jails or prisons, state-based organizations, advocacy groups, et cetera. It could be the community. And the community is a broad term that you need to -- if you're approaching a study thinking that you'll partner with the community, you have to define what you mean by the community and that needs to be something that -- a group that makes sense to partner with depending on what your strategy is. So, obviously, strategies that are implemented at community level or that target the community are ones where it most often makes sense to have the community as your partner and bringing that up with, I guess, not a note of caution, but just something to think about realistically. Working with the community is a very broad base of people to work, and so you need to think about for your questions do you need a partner that broad and what the challenges will be in partnering with a group that's that broad. Now, we're going to talk about some of those challenges later in this talk. So when you're thinking about the type of organization or the type of partner that you're wanting to work with, you need to think about how do you really determine if you're not building something with some group that you already have a relationship with, if you're going to be starting new with a group, how do you gauge first by topic area who the group should be? And then among the possible clinics, let's say, that could be your partners, how do you determine what's really -- what are signs of sort of positive potential in who your partner is? So, obviously, you want to have a group of people who have some interest in improving the health of some population. They need to have an interest in the work that you're doing and they need to have an interest in working with you and that their interest in the work that you're doing is probably, you know, growth of their relevant knowledge or experience in the area, and that's important because that's -- their knowledge and experience are one of the things that will make a very valuable contribution to informing your research. But they also really have to have a realistic idea of what partnership with you will entail, partnership in a research project and possibly longer term partnership than just one research project. So in order for you to really achieve that kind of understanding with your partner, you need to have a clear idea, at least to begin with, of what kind of partnership your seeking, and then, of course, because it will be a partnership that will lead you to some kind of negotiation with them of what kind of partnership they want to have with you. So, in thinking about what kind of partnership you'll have, you need to think about what defines a partnership, right? So, obviously, it's the description of the kind of interaction that will be going on between two or more entities. In the case of research partnerships, that can range from something like having community partners as an advisory group. That's very common. It's pretty much in the most of services research. That's something that's expected that you'll have, advisory groups helping you understand, working in your community or in your setting. But it can extend to sort of a relationship in which the researchers and the community partners are on a closer to equal basis, or it could also extend to efforts to try to really have an equal partnership between the community partner and the research partner. Those are challenging and that's -- has some specific requirements that we'll talk about. But the think to keep in mind when you're talking about the research community partnership is while they're many strategies and when you read the community-based participatory research, of course, an ideal that people have to try as much as possible to not have a power differential or not have a hierarchy between the research and the community. But the truth of the matter is particularly since you're engaging the community group in research, which is your purview, it's hard for there not to be some form of power differential, and that's something to be mindful of in thinking about, you know, areas where you want to minimize that power differential or just acknowledge that power differential. So places where that differential can show itself is what happens with the money, the support that you get for doing the research. What is the differences in education between, you know, the actors in this partnership between the researchers coming usually from an academic setting versus the education level of the people that you partner with, and coupled with that is the different styles people have of interacting with each other, of interacting within their own setting and interacting across settings. And so thinking about that power differential, it brings up the question of whether -- how much interest people have in striving to have a community-based participatory research kind of relationship or not. And I think -- as I said before I think it's an ideal that many people strive for, it's also something that's very challenging to accomplish. So let's talk about CBPR, okay? There's a lot that's been written. There's some great resources online including educational modules and things to help people understand how to do or strive for CBPR approach. But some of the main components of it are that -- you know, it's strategy for engaging community members in the research process, community as you have defined it. And it has as one of its goals employing local knowledge in understanding health problems and in designing interventions and also, depending on how rigorously the model's applied, not just designing interventions but conducting the research, looking at the results, evaluating the results, disseminating the results. And that's where the goal is to invest community members in their processes and products of the research. Now those are all bullets that can apply to lots of kinds of community partnerships that aren't really necessarily exclusive to CBPR. The things that are unique about a CBPR approach and are especially relevant when you're thinking about applying to NIMH or other NIH institutes is the model as its developers conceived of it was that the goal is really power sharing between the community and the research side of the endeavor. And the -- one of the outcomes of that is that -- not that just that the research is done that but one of the products of research is some kind of concrete action that happens as a result of the research and underlying this are a lot of political ideas about, you know, the -- what it means to try to eliminate the hierarchy and empower the community in terms of influencing the research. So those are good goals but challenging goals and goals that are sometimes challenging to successfully incorporate in applications to NIH. And in particular, if you are thinking about submitting an application or you're saying you're using a CBPR approach, it means that your application is going to be reviewed by CBPR reviewers. And I have sat in on and observed, especially from the health disparities portfolio, reviews of applications that state they're using the CBPR approach. And I can tell you that CBPR reviewers have very high standards, as they should, for assessing their particular approach. And I think we see that also when we have economics reviewers who review, you know, economics grants and qualitative reviewers reviewing qualitative grants that people have a motivation to have the principles of their area be adhered to well. But that -- what that means for CBPR in particular is I think there's a special challenge in that CBPR, it isn't as easy to evaluate whether CBPR is happening or not as it is to say someone has a well designed, randomized control trial. But there are different opinions and different standards from the folks who review what meets the goals or doesn't meet the goals. And so I think that -- I'm not necessarily discouraging people from using the CBPR approach per se, but I want to -- when we're talking about all the possible sorts of partnership studies you could propose for you to think about what makes sense for your study, what you think will -- has potential to be successful in review and what you can really successfully propose yourself as being capable of accomplishing. So in that approach, of course, going back to what -- the core of what will be looked at in an application to NIMH is what kind of partnership do you need in terms of, you know, answering the question that you say you're trying to answer? And what kind of partnership do you already have if you already have a partnership with some group? And so you want to think through, when you're thinking about proposing a partnership, what are the areas of mutuality and what are the areas of hierarchy that you anticipate existing between you and your partner? You want to think about, for example, as you work as a team and how ever you are interacting, how does it work for setting priorities for the research project and for the daily operations of research activities? How does it work in terms of decision making? Things are likely not to go exactly as planned in your study because they almost never do and in addition people are -- things are likely not to go exactly as planned in the day-to-day setting of your community partner, and so when challenges arise, what is going to be the decision making strategy and who -- sort of who gets the final say? One of the core tenants in CBPR is sharing power and that includes sharing funding. So they're -- in order to do a study, you'll be getting some set of resources, some financial, maybe some others are in kind resources. And so the question is, how do those resources get divided between you and your partner? Another issue is how much entree each partner has into the other's setting? So normally when we think about a resource partnership, it's much more typically focused on how do the researchers get into the community setting or how do the researchers get into the clinical setting? Is that going to be a two-way Street in your partnership? For example, there's some thought to people in the community partner side having access to library resources at your university or educational resources at your university, and -- along with the decision making is sort of at the level up from that, is how is conflict resolution going to be handled with isn't agreement between the partners, how does that get resolved? So for each of these things, there's no necessarily recommended or indicated goal that you should have per se. So it isn't the case that you're going to be successful if you have a 50/50 split in your funding. What you need to think through is what you think you can strive for in your partnership, what the current state of your partnership is capable of sustaining and what you negotiate with your partner in terms of what you need here is to have a mutual understanding of all of these issues. And with the also understanding that -- well, let me go to the next slide. Yeah, I think I might have already said most of this, right? But just to reiterate it a little bit. So what makes sense in terms of the research question, but also what reflects your actual relationship? What you're capable at the moment with the partner that you're working with. There's recognition that these -- for most kinds of productive partnerships, but specifically for research community partnerships, they don't get bore and fully formed. They take time to build experience to develop trust and to develop mutual understanding of what's going on. And so when you propose a partnership in your application, you're going to be describing in the application what your partnership is, and you're going to want to be able to convey with the reviewers evidence that your partnership is at the level that you're saying it is. So things that you can build into your proposal that would be evidence of that would be work that you've done with the partner in the past that shows, you know, working together and coming up with products and goals successfully. It could -- of course, the letters of support from your partner are important. Those should be, you know, letters that are written in a way that demonstrate your partners understand what the project is and what their role in it will be. And keeping in mind just that whatever you say in the application, all the reviewers will have to go on in assessing it is what you write. So you need to think about what concretely you put in there. And then finally, you need to think about, in what you propose in terms of partnership, what the funder values and what the funder is willing to support. So there's been a history of NIH interest in supporting. First, it was community-based participatory research explicitly, and I think over time that has sort of evolved to something without as high a standard, but studies that foster partnerships with the community. And as I said, in terms of services research, this is the place where if -- our idea is about getting interventions and services used and integrated into community settings, we, of course, need to know the community. But there's the question of how much support from the grant or from a support mechanism can be used to work on developing the relationship or taking care of the relationship versus how much of that partnership is something that's considered a positive but intend -- expected on the part of the funder to have already happened and not be happening with their money? So their previously had been an announcement that was first issued across NIH, and NIMH was signed onto it, that supported CBPR. And then NIMH issued its own CBPR announcement and both of those announcements an R21 which is money for pilot work, pilot studies, that allows for some part of that money to be -- in time to be used for developing the relationship with the partner. And that was intended to kind of set the stage for folks coming in for the R01 level announcement. That would allow them to actually use that or implement that partnership in the course of the research study. Those mechanisms existed for a while, and at the moment, NIMH is not offering or signed onto either of those. When I went and did a search in what is listed as NIH grants, there is -- it's still an announcement from the Office of Behavioral and Social Sciences Research that supports partnership with -- I can't -- now I should have written it down. I can't remember exactly the title, but it's partnerships with communities that are -- have challenges or barriers to access to services. NIMH is not signed onto that. The other institute that is offering something that is explicitly CBPR-specific is the National Institute of Child Health and Human Development, NICHD and what they're offering is a meeting group, I think it's an R13, to work on developing the partnerships. Now, among foundations and other kinds of funders, there may be more money available to kind of support the relationship development part of a line of research that you might be thinking about building. But in terms of what our funding could support -- our current -- the way our funding is currently structured, a large part of your -- the beginnings of the development of a partnership is something that would be happening prior to submitting the application. So that's the thing you need to know in terms of when you build the relationship is you need to understand what the expectations are of the entity providing the support. So there may be times you can do it with funding. There may be times you need to do a lot of it before you actually apply for funding. But in either instance, maintenance is going to be ongoing once you start the partnership, and it's going to be something within an application like an R34 or an R01 or an R21 that maintenance is probably integrated into the activities of what you're doing in the course of the research without it necessarily being something that is specifically funded by the support. So thinking it -- with that as a backdrop and thinking about you have a research idea that is amenable to -- that makes sense for there to be a community partnership and you have a clearcut idea of what the partnership will be like and you even potentially have a partner who thinks they want to work with you, there are things to things think about in develop -- in terms of developing this relationship. And some of the things to think about really has to do the differences and similarities between community settings and academic settings. In each case, people from either side of this dichotomy are often coming from settings that are -- where everyone is overworked and under-resourced. And the dedicated people are really working overtime, possibly not for much pay or any pay. But in either setting, you have a group of people who have unique skills and unique experiences and potentially not overlapping and also have their own jargon and their own way of understanding what they do and how they do it. In each case, the work is often complicated for you folks who are interested in suicide prevention, and you know, mental health crisis in general are very complicated and stressful environments both on the research side and on the intervention or provision side. And another potential dichotomy between community and academic settings is that the accountability structures, the productivity expectations, the timelines and calendars and bottom lines can be very, very different and just -- and in addition if you're starting brand new with a community partner, you may be working both on your side and on their side with the group that's really not used to working with each other. But taking a step back for a moment and talking about the timeline issue, something that is a very common problem in folks applying for applications with us and going through our review process and then often going through their revised resubmission process and another round of review is -- often underscores how big a difference there is in funding timelines between community settings and academic settings. Notably, a community setting, say it's a clinic or some other entity that is used to getting funds of some kind to provide their services or to conduct their activities, are most often used to state money or county money or maybe even foundation money which typically has a much quicker turnaround time from when you submit, when you find out if it was success and when you get the money. And I don't know how much you heard about our timeline or you may know from the places where you are employed, but our timeline is much longer than that, but it's often, you know, nine months to a year from the first submission before you find out whether you're going to get funded or not. And there are long periods of indefiniteness, sometimes even after the score on your application is known. And if it -- you know, if the way it's going is for a revised resubmission, you're going to have to go through it all again, and that can be very difficult. I've had folks calling me saying, "Our partners can't way. They're going to bail from this partnership -- you know -- they were really, you know, ready to go on what we had setting on and maybe depending on some of things we had agreed to and they can't keep waiting." And that's a really big challenge and purging the community and academic settings and one that really helps if -- as you're entering into this that you educate your partner about what realistically our funding cycles are like and that -- so that they understand it. It may not make it that much better to have to live through it, but that hopefully can work to lessen some of the frustration that the community side might have in how NIH proceeds very slowly. So you might want to also think about -- and it might come up in the process of you negotiating your partnership -- what is in it? What is in it for partnering with you from the perspective of the community partner? Why would they want to do it and what would they expect to get out of it? So potentially participating in some kind of a research project might give a setting, a provider setting or an advocacy group sort of partner new understandings about their programs or new understanding about their clients or their members. Ideally, we're talking about interventions and services. You think that the setting might be thinking that, you know, partnering with you could really help improve health outcomes for their clients. And improving health outcomes in the context of research is a tricky question, right? You need to be clear and upfront about how realistic that is in the course of what happens during your study, how concretely in the course of say pilot work that you're conducting via an R21 or an R34 is going to improve health outcomes in a setting. Usually, we would think that wouldn't be happening at the pilot stage. That there'd be some hope about that happening at the R01 stage. But, you know, you might get feedback from your partner saying, "You mean after we go through all of this for three years, at the end of this, you won't be able to tell us what we should be doing differently with our clients or what I should be doing differently with my family member?" That's something that needs to be upfront because that might be something that the partner is expecting to get out of your relationship if you want to maintain and take care of that relationship during the course of the research so that your partner stays involved. Other things partners might be thinking about getting of the relationship would be things like money, prestige or influence that might come to their organization from being involved in research, you know, federal research, NIMH research might have some cachet. And that might -- that does bring up another sort of a level of things to consider and that is money, prestige and influence might be something that are goals of the leadership of particular organization or group that you're partnering with. Those things might not be as important to the frontline folks that you might be dealing with who may not have the same investment in wanting to bring prestige to their organization. They may have very different kinds of day-to-day issues and concerns that -- for which the prestige brought to their organization is not really a priority. Similarly, an organization might be interested in partnering with you to enhance their site's capacity for research, evaluation, data collection and analysis. Sometimes these partnerships allow, especially clinical settings, to develop new data systems that really enable to do a lot of things and potentially obtain other support on their on, not just through you. And again, those are places where enhancing a site's capacity for research might be a priority for someone at the management level, but someone on the frontline staff level might view this as creating of greater workload for me in terms of collecting data that feeds into the research, that builds this prestige, that that person doesn't really have as a priority. So these are things that, you know, you need to think about and talk about in your discussions with your partner to understand what their expectations are and dispel ideas about unrealistic expectations. On the other side of the coin, what is in it for you to partner with the community partner of some sort? Personally I think one of the biggest pluses from a community partnership is the potential for innovation. And I think one of the things that there's often criticism about is, you know, NIH is supporting Ivory Tower research. That doesn't really have anything to do with the issues of, you know, quote "real world concerns." But also you have the thinking -- the experience and thinking of people with a very different perspective and a very different set of experiences that can actually bring something really exciting to the way you frame a research question and think about a line of research that you're trying to develop. And that would come from a greater understanding of community issues and settings, what the issue and priorities are of the folks in those settings, consumers, family members, providers. You may be developing a services package or an intervention, but some day that will leave the door, ideally, and actually be embraced and used by people in the community. So if an idea from -- that comes actually from engineering to get the endstage user's input on how you're developing your tool or developing your -- what you have to offer. And this is a place where not just innovation, but potentially greater relevance of what your product will be in greater usability. Similarly, when you're getting this input from the community, it can help you start early on to really start to build in things that will be important for the ultimate dissemination and implementation of your intervention strategy. We don't want to -- I have a story from long ago about some folks who were interested in developing an intervention where case managers for -- who were working in community settings with people with severe mental illness. Their idea was these case managers should be providing safe sex education to the people with SMI so that they can, you know, lessen their risk of contracting sexually transmitted diseases, especially AIDS. So what they did was they got an R21 to do some pilot work and they went and talked to consumers, case managers and case manager supervisors about this intervention that they were thinking of developing. So the obvious questions that they were asking were, you know, "Would you be willing to, you know, either receive this or deliver it, participate in it? Would you feel comfortable talking about these issues? Would you -- you know, do you think this could happen?" And, of course -- or maybe not so obviously -- the consumer said, "Yes, we think it's a good idea. We think we could do it. We can talk to our case managers about it." And the case managers said the same thing. When they interviewed the managers or the supervisors, the response was, "Yes, we think it's important, and yes, we think it should be done, but no, we do not think our case managers should be doing it." And reason why was that our case -- their case managers are already very overextended, they have gigantic caseloads, they already have more than they can handle. And the manager's perspective was, "We would not endorse this because our people don't have the time to do." This feedback enabled our applicants to rethink what their intervention was going to be. And so rather than developing a package where they trained and educated case managers to deliver, you know, the safe sex education is a pretty well established package of what gets conveyed, and they were looking at adapting it for how you need to deliver it to people with SMI and how you need to have -- you know, how you need to have case managers prepared for delivering it. This feedback from the managers told them, "What we need to do is have a training that trains people in safe sex education settings to be able to deliver this to people with SMI and have then a way that case managers refer their folks to those settings." So that just sort of shifted the point of delivery. But that was productive feedback that they got from actually asking people in the community, "How do you think this would work?" So, this one as I saw it here at the end of the slide is, you may also be thinking that -- as you're building the relationship with the community that this is just one step in the line of future research. You might also have an interest in this future research being a setting for training future students or future fellows. And so I'm going to -- looks like I'm blabbing on and running out of time, so I'm going to just run through these challenges and some of this has already been covered. So the challenges in having and keeping a relationship is, you know, you got to develop that mutual understanding, you got to find a way to look at and address and differences in values and priorities. You may or may not encounter real conflict between treatment and research. An example from a grantee was their working with a partner that provided services to street kids, homeless use with mental illnesses, substance abuse problems, they were completely opposed to randomizing. It was just countered to their culture and countered to all of their efforts to try to engage kids in treatment to say, "Okay, we're going to get all these kids and we're going to treat half of them, at least to begin with." It was something that could not happen in that setting. You might find in the course of conducting your research that, you know, it becomes obvious that there are working styles and different time demands across the partners. You may get things all set up and have everything going along way and things change. There's staff turnover. Many things can change on the community side and on the research side. So keep in mind that partnership is a process, not a structure. It's something you'd be working on all the time with your partner. A partnership is an approach to research, but it is not a research method. It's not a replacement for the things that need to be in place in addition in your study in order to be able to answer the research questions that you want to answer. In thinking about this, there's some literature but a limited literature in terms of real life examples that you can use as a guide, and of course, because you're developing these relationships uniquely with -- you know, between you and your partner, there's only so much that can serve as a guide but -- aside from broader principles. And while you're in the course of working together in your research partnership, you need to have some -- pay some attention to and figure out some ways of gauging the health of your partnership, gauging how well it's going, how do you know when things are going wrong or how do you know when things are going right, and what are the kinds of outcome measures by which you could gauge your partnership? There isn't -- as I said, there is some literature out there and I think if you're interested in pursuing this, you ought to read it, because it's pretty interesting. But there isn't an agreed upon answer to that question. So another thing to think about in terms of challenges is your -- bringing your research staff into the setting of the -- of your partner. You need to think about to what degree the research staff is really -- has capacities of not in terms of building relationships and working with partners and vice versa. The capacities are limitations on the partner's side of being able or willing to understand and work within the research structure. There's team building that is going on all the time and there's a potential in the community setting that you're entering when you have people, staff or others from an organization selected to be your partners in research or be -- to be the data gatherers in research or other -- what other capacities they function in your study that the line between who's an research insider and a research outsider can be blurred. And the question about to what degree people who partner more in the study get rewarded more or advantaged more by the organization can create some challenges within the organization. So the little thing here says I have 34 slides but I really only have I think 19 or 20. So we're almost getting to the end and I see there's some questions. So if you want to get started on building a partnership, you need to do your own -- you need to do your legwork on who are the possible partners in your community. And among those possible partners, you need to think about what are the current issues of concern to those partners? But the thing to have a partnership doesn't mean the researcher is bringing all of their agenda to some setting and hoisting it off on the partners. I think you're not going to get ongoing partnership from folks if there isn't something in it for them, and you need to understand what might be in it for them. Some people go so far as to do key informant interviewing with different organizations to really get a sense of what their interests are. You -- all of these things, of course, are building toward you getting an organization or an agency to give you entree, to actually let you come into their site, be exposed to their staff and their clients and understand what's going on there and understand -- start to hammer out how you will work together. So how are -- you will work together is a -- you know, a process that gets built through trust, communication, connectedness, work efforts that are meaningful to everyone who's being asked to do work and meaningful products. This is going to require open communication, fairness, a meaningful planning process, and this is a component of the infrastructure of your study, the operations of your study, and a component of your relationship that may even expand outside the study. So it can be helpful to go -- to make it so formal as to develop operating procedures and bylaws that are things that are agreed on by both parties. The citation that you see at the bottom of the slide is I -- the curriculum that's on the web. It's for CBPR, but I think there are a lot of concepts that you can pull from it that will be useful, not just for CBPR but lots of kinds of partnership activities that you want to pursue. And in fact, by the way, if you are going to pursue CBPR, you probably do need to develop operating procedures and bylaws and convey that to the review committee. So just some other things that people talk about why it's good to take a CBPR approach, that there's an idea that this will improve informed consent, improve recruitment and retention, improve cultural sensitivity, improve interpretation of findings, increase relevance, increase implementation, improve quote "success." You don't want the make claims about any of those things in your study that, "We're using this model because it's also going to result in these things," unless you have some evidence that that is the case. And things to think about again from the partner's side is improving recruitment and retention in your study is obviously a goal of yours. To what degree that's a goal of your community partner is somewhat of a gauge of their interest -- I keep knocking this thing -- in this research. But don't -- when you're trying to make the argument for the kind of partnership that you're going to have, don't oversell it. That's my bottom line. Don't make claims that it's going to do things that it can't do. I can tell you in services research and I think in the interventions research side of things here at the institute, we think partnerships are a good idea. They just need to make sense in the context of answering the research questions. So just don't oversell it. That's -- and then other things to think about when you're thinking about what your partnership is, who owns the data, what happens with the data, how does it get interpreted, how does it get evaluated, how much mutuality are you going to have with your partner in terms of those kinds of things, who benefits from publication of the data, what do you do with negative or unflattering findings? Those particularly can have a lot of potential risk for an organization that maybe was implementing some kind of an intervention of a services package if the research shows that they didn't do that very well or it didn't lead to very good outcomes that has implications for that organization that need to be thought about ahead of time. And, again, these are things that there's not sort of a recipe for you to follow but that need to be thought about and worked out in the process of your relationship. And so with that, I leave you with my e-mail address if you want to talk further or ask questions. And I see that there are some questions and so how does this work now, Jeff, in terms of -- do I just scroll through these and -- Okay. Two questions, okay. Ooh, a long question, okay. "Given the economic strain that many community-based organizations are facing, restructuring, loss of fundings or reductions in force, how do you I sell them on the benefits of research? For example, there are many factors that influence mental health service use. The behavioral model of mental health service use is commonly used to examine general health service use and has been applied to mental health service. However, it does not account for stigma as one of the factors. So a study proposing that stigma should be considered as a factor would be useful. However, an agency rep may wonder, how can you associate these general feelings of perceived stigma with actual service use? How can I address that?" Well, I can try. Stigma is an important question but also a difficult question both on the community side -- and by agency reps, I think you might have meant someone representing the community agency. But those are also challenging from the NIMH side of things as well because stigma is a challenging thing to measure. Stigma can happen sort of community level stigma, sort of the like the broadly held ideas people have about people with mental illness, wrong ideas by the way, being, you know, dangerous, unemployable, et cetera. It's hard to design a study that could be funded by an NIH institute that would implement a community level intervention on that and then really actually have a measurable impact on first reducing stigma in the community and being able to tie it to what you did and then a really important component from our side is did reducing stigma enable people to actually -- more people to receive services? So I guess the -- probably if you're thinking about doing an inter -- thinking about an intervention that addresses stigma, it's probably a good idea for you to contact me and for us to talk more about it because I -- there are just many levels of challenges on our side having to do with stigma as well as on the agency's side. And the second question is: "For the key informant interviews, will this include a needs assessment? If so, can you suggest some tools and measures for this interview guide, et cetera?" Well, when I was talking about key informant interviews, it was more from the perspective of figuring out who you could partner with. And I think that then the kind of needs assessment you're talking about would be a second step in that when sort once you have more narrowed down ideas who you might be partnering with, their agency or organizational needs would be important for you to consider in terms of what are their values and priorities. But remember, you also then have to kind of tie that back to what kinds of research questions you might be able to ask and answer. So off the top of my head, I don't have tools or measures to suggest for that or an interview guide. It's another place where it's probably a good idea for us to talk offline more specifically about the kind of agency that you're thinking of working with and what kinds of challenges they might be facing. You're welcome. Okay. And I see here -- oh, someone left. Okay. So there's one minute left. Does anyone have anything else they want to say or ask? Oh, here. "How do I go about balancing the clinical demands of the community center versus the research needs of my institution? For instance, I have been approached to provide PTSD treatment for refugees in community settings and they want me to start now, whereas developing a research study around this takes time." Yes, that is a challenge. And one of the things to think about -- I mean, being able to start now could probably go a long way toward helping to develop and solidify your relationships with the community partner. And so something I didn't get into, but something that is an important thing to think about is in developing your relationship and keeping it going with the partner, there may be times where you have to volunteer your time. You are a valuable resource, you and what you know and what you have access to are a valuable resource. It's -- those are probably things that the partner sees as something that would be good for them to have access to you and what you know and what you can do. And potentially you -- in this instance, what you're describing, you -- it might mean that you're going to be doing some things unfunded or unfunded by NIH to get started. And it isn't a bad idea, actually, to have some experience in working in the organization to start to understand things that might inform your research questions. You could also think about getting small about of seed money from your institution, your department might have money, a small about of money that they could give you to get something started in that setting. They might also release you from some of your time that's committed at your institution, maybe some teaching time in order to invest in getting you going in the community partner. And you should also, if you're entering in this process, start educating your partner about what the ultimate timeline is going to look like for research study to get started. Anybody else? Yeates Conwell: This is Yeates Conwell. What you said was really masterful. Thank you very, very much for joining us today to provide that. This is really complicated stuff. I wanted to I guess just make one comment. Denise Juliano-Bult: It is. Yeates Conwell: For our future discussion as a group -- and maybe just get your very reflection on it -- it takes a considerable amount of time and energy to develop a partnership, nurture it to the point where it's going to be helpful for research. And that timeline really needs to be understood for early bird investigators because you have timelines yourselves with regard to promotion and tenure. And you know, in certain ways necessarily the best way decision to begin a new partnership, it may be particularly if that partnership is embraced by the institution in which you work. We've talked about academic development plans and the importance of matching one's own individual trajectory vision to the strategic priorities of the setting in which you do your work. And I think that's just particularly important with community partners, community-based participatory research because the investment really needs to go beyond the individual both in terms of the time that you have to dedicate to it but also the commitment that the institution we typically work in those academics have to make in supporting us in our partnerships. Denise Juliano-Bult: Good points. Yeates Conwell: Okay. Paula had a comment there. I'll just read that and then we'll wind up here. But just collecting quantitative data on the incoming subjects and getting them to agree to a follow-up might be a beginning -- in beginning research projects and just the standard quantitative interview.Reactions to that, Denise? Denise Juliano-Bult: Okay. Well, I see that -- I have my e-mail address there, so for anyone who had wanted to follow-up questions or to talk about anything further, I'm happy to do so. It's been very cool using this technology, and I liked it a lot better than I thought I would. So I think that the way you're structuring this VMED overall is a pretty cool technological thing that has a lot of potential. Yeates Conwell: Thank you very, very much. So this presentation will be up on the web site and we -- I'm sure will refer to it again. Denise Juliano-Bult: Okay. Thank you all. Bye-bye. Yeates Conwell: Bye. Yeah, we have another piece of our agenda for the next 25 minutes which is Amanda. Are you with us, Amanda? Amanda Garcia-Williams, who is a doctoral student at the Rollins School of Public Health. Take it away. Amanda Garcia-Williams: Okay.So, yeah, so I'll give you background just briefly before I go into this. So as he said, I'm a public health doctoral student. Hopefully I'll be in candidacy in November. And my degree is behavioral sciences and health education, so it's sort of like a combination of social psychology and epi and kind of all that smushed together into a big pile. Okay. So my intellectual focus is injury prevention with a specific focus on intentional and self-directed injury, so suicide prevention, but I've done some work in intimate partner violence. And at this point, I'm starting to slowly expand to other forms of injury prevention such as like motor vehicle or helmet use, things like that to sort of be a little bit more broad in my focus. And the way I've written my ADP is with sort of two trajectories in mind. Since I'm doing public health, there's sort of two avenues that I can go down and one is within the academy and the other one is outside of the academy. And a lot of the things that I'm going to -- that I have within my ADP, kind of work towards both of these trajectories, so when the time comes that I'm out of school, I'll be able to sort of decide which way I want to go. So my trajectory number one is to become a research professor within the field of injury prevention. And I'm going to say injury prevention broadly, again, just to sort of -- I'll talk sort of about my intentions regarding that. And then my trajectory two is to become a senior social science researcher in injury prevention within a government agent such as CDC but also, you know, something like a state health department, the VA, DOJ, something -- or DoD, so something sort of along those lines but not quite in a university setting. And so I have a couple of main career development objectives that are specific for both trajectories. So the first one is obviously to complete my doctoral dissertation by July of 2015. The second one is to network with injury prevention researchers in Atlanta, and I'm going to kind of really do this by attending meetings held by Emory Center For Injury Control which attracts sort of the main -- or the -- and the primary injury prevention people in Atlanta from places like Carter Center or Georgia State, the CDC, it's sort of this meeting center for all these people. And the third thing is to engage in the didactic research mentorship and other activities that I'll describe later in the ADP. And then my objectives specifically for -- if I decide later on to go sort of a professor, research professor route, these would be to network with faculty that work in injury and violence prevention at a university in Atlanta, Georgia State University, to identify possible -- and out examiners, mentors. And I've identified a couple of people that may work. And so now it's just a matter of talking to them and start figuring out some ideas. And the second objective is to write a postdoctoral NRSA with one of those faculty members. And I would have to submit that by the dates here. I also plan to write an ASSP postdoctoral fellowship with either a mentor at Emory or Georgia State University. And so this one would be -- I don't really have an idea yet or what these postdoctoral fellowships would be just yet. I'm still kind of -- need to start thinking about ideas, but these are the mechanisms that I have in mind. And then I plan to write an NAH loan reimbursement application to accompany my ASSP postdoc application. And I'll also write -- plan to write one of these if I get a postdoctoral NRSA. And then finally I'm going to write for an SBE minority postdoctoral fellowship through NS -- International Science Foundation. And so these are the main mechanisms for postdocs that I've found that I -- that would work. And I have a question down here that we can talk about later. But just to keep in mind in of other postdoctoral mechanisms. And also I have some questions about how to R99 or R00 works, the sort of postdoc that transitions into a professor type role. And then my career development objectives for outside the academy. This is to expand my expertise in unintentional injury. And I'm going to be doing that by working with motor -- vehicle safety data associated with another of my mentors, grants, she's my faculty advisor, and so she has some data, and her interest is in things like graduated driver's license for new drivers and how that impacts mortalities. Another objective is to conduct a brown bag seminar at CDC in the injury prevention division. So I plan to do a brown bag with my dissertation work to see their feedback on it and also to give them a sense of what I'm working on and where my skills are as this would be a division that I'd be interested in working in. Then I would network also -- network with injury prevention researchers outside the academy, and it would include people such as Drs. David Fleet and Alex Crosby at CDC to identify postdoc mechanisms at CDC or other ways that I can kind of have that postdoc experience outside of the academy. And then finally I'm going to apply for EIS in fall of 2014 as well as applying for the PMF or Presidential Management Fellowship program. In fall of 2014. And so then in terms of my description of my autobiography and my critical self-assessment, again, my interest is -- it's -- my primary interest is intentional injury and that started in spring 2009 and I'm currently working on expanding. And really the primary reason for this is to increase my future marketability when I graduate. And also to prevent burnout because working in suicide prevention is not totally easy, so -- in terms of just the content area is little bit heavy. So having sort of a broad array of research interests I think will prevent me from fatiguing on that topic area. And then my context and direction has been pretty varied, so I've worked in health care quality, I've work in depression and epilepsy, and then currently, I'm working in suicide prevention. So I've had a lot of different areas that I've worked in. And my main strengths is theory-driven research, qualitative research methods and more broadly, just research methods as well as program evaluation and public speaking, although my public speaking at the present moment might feel leaving a little bit to be desired. And then my weaknesses are quantitative data analysis, intervention mapping and design, so creating my college education programs or specific interventions for something like an RCT or something like that. And other resources include manuscript writing, grant writing and community needs assessments. And so my description of my faculty members, so through VMED Dr. Nadine Kaslow is my faculty member but she's also my sponsor for my NRSA and my dissertation chair. And so she's going to serve as both a process and content mentor for intention injury prevention in the academy trajectory, so she's a full professor and is -- really knows, you know, all about how to, you know, become a really success academic. She's a really successful academic. And she's an expert in intentional injury prevention. So she's sort of the ideal person for this. And then my faculty advisor, Dr. Nancy Thompson, who is also a member of my dissertation committee and a mentor on my NRSA, she's going to serve as an additional content and process mentor for unintentional injury prevention and nonacademic track. And although she is currently a professor, she also has worked at CDC for a long time, I think ten years she worked there. So she has a sense of the pros -- not the pros and cons, but what they're looking for in terms of marketability as well as she knows a lot of people still at CDC so she can be very helpful in providing me with networking opportunities. And so both of these mentors are going to help me with networking. And I'm currently working closely with those women on their projects, which will give me practice with quantitative data analysis, grant and manuscript writing. And there's a lot of overlap between these two mentors, so Dr. Kaslow is not -- you know -- doesn't yet -- it's not as so Dr. Kaslow doesn't know anything about -- not in her academic track or Dr. Thompson doesn't know anything about academic track, it's more that I'm sort of using them as -- together to get to these sort of -- to meet my objectives. And so my plans and activities in terms of clinical care, that's really applicable to me since I'm not -- that's not my degree. But teaching, I'm not going to engage in anymore teaching. I've been a teaching associate and I've had my own course in public mental health at a local college at Agnes Scott College. And -- although these experiences were good, they did provide me with insight into my desire to not teach, so understandably that means that I would not be doing a full professor type job. It would be more of a research professor job in the future if I stay in academia. But teaching is something that is -- it's not -- like I don't see it as part of my long term sort of career. And so mostly I'll focus on scholarship and research. And so I kind of gained a little bit of percentages of what I'm seeking, like the amount of time I'm going to engage in all these different activities. So in terms of didactic, like my coursework, although I'm passed taking coursework in my degree program, there's some things that I think I can take that will be helpful. And this includes ethics, a course in criminology, which will sort of boost my knowledge of intentional injury beyond suicide prevention. Also taking a more in depth course in qualitative data analysis and a course in in depth interviewing, a course in grant writing, and then there's a class on -- it's called Death and Dying but it talks about death and dying from sort of a nonclinical sort of perspective and more about the way the different cultural groups understand death and dying. And so I think that that would be helpful to both intentional and unintentional injury in terms of preventing something but understanding how people conceptualize that, sort of the worst outcome of intentional and unintentional injury. In terms of mentorship, I'm planning on doing a directive study with Dr. Kaslow in the spring. And so it's good for me to practice intervention mapping for suicide prevention programs. So I'll use the techniques outlined by Dr. Bartholomew to do -- to create sort of a pretend suicide prevention program for college students, to give me practice looking through the literature and creating an intervention that's evidence-based and using sort of intervention mapping techniques, which I haven't had the chance to do. So although this is not going to be used, I think it'll be a really good experience just to practice those and get Dr. Kaslow's input in terms of usability and, you know, what to think of from a more clinical perspective since she's a clinical psychologist. And then I plan to work with Dr. Thompson to do quantitative data analysis of her injury data. So, again, that'll be sort of doing directed study. And in terms of research, I plan on working on my mentors vary -- mentors' various projects. And this will include data analysis, grant and manuscript writing, so that'll overcome some of the weaknesses that I have in my training. But I'll also work on my doctoral dissertation, and this will give me a lot of help with both grant and manuscript writing, qualitative and quantitative data analysis. And so, again, this will give me more areas to practice and to bolster my weaknesses. And then sort of other activities to meet my objectives are to attend relevant lectures in Atlanta and these include lectures on ethnics and grantsmanship, injury prevention, but also listen to injury prevention webinars that are available online and then attend conferences such as AAS and APHA. And then just -- I'm not going to go into a great detail about my dissertations since I want to have more feedback, but just to give you a sense of what I'm going to be practicing some of my qualitative and quantitative research techniques on -- to increase my area of weakness, my dissertation is suicide behavior in college students and peers response, and it's a mixed message study -- actually I skipped passed that to here. It's a mixed message study that's kind of in two parts. One's going to be in depth interviews with undergraduate college students who've had previous experience with suicidal peers. And I'll be using a social psychology theory of arousal cost for word to understand how they decided to help their peer if they did and ask specific questions, but are framed by that theory and the constructs within that theory. And the second part of the study is I'll be doing a web-based survey. So I'm going to determine if the arousal cost for word theory is useful for understanding college students helping behavior towards suicidal peers. So I'm taking a random sample of 2,000 students. And they're going to be given one of two survey instruments at random. And so I'll -- we'll use mediation analysis to see if this theory is helpful is understanding the likelihood of a peer helping -- or of a college student helping a peer who appears to be suicidal. And in terms of my funding, I'm currently funded through an NSRA training grant for three years. And then that doesn't pay for my research costs, so I've gone -- I've already engaged in some grant writing to help boost that part of my -- or that weak area. And so I've received a scholarship through the Emory Center For Injury Control and a grant through the American College Counseling Center. And I'm applying for two more grants from SOPHE, CDC and one through APA to help pay for the remaining costs associated with my qualitative study. And so this is my final slide. It's really the considerations about developing the ADP. It was actually really hopeful to develop this, but it is also extremely stressful and caused a lot of sort of anxiety and worry about the future. You know, it required a lot of thought in terms of what I really wanted to do, and so I really developed with ADP with the idea of how can I be as marketable as I can be following graduation. And part of that is because I have some geographic limitations in that I need to to stay in Atlanta. And so I really want to figure out a way of making myself super marketable so that I can find something that is -- is what I want to do but that helps me stay in Atlanta or possibly, you know, nearby, like Kennesaw or something. And another big goal of mine, which I don't think I integrated very well in the ADP was -- I really want to win -- or not win -- but apply and receive an NIH loan reimbursement grant, and so I really was trying to think of strategies to make this a possibility of ways that I can develop, a postdoc that is good for me sort of career wise but also something that would be really competitive for the loan reimbursement grant. And so that's a really big goal of mine. And overall, I was challenging writing this because I have interest in both trajectories at this point. It's not that I want to become an academic more than working at some place like CDC or the VA. Those are attractive to me at this moment -- at the present time, and I think I'm going to use sort of -- when the time comes whatever works out or work out and that'll be the career path that I'll take. I'm going to try to use state as sort of my deciding factor when I graduate. And also in terms of the consideration of what I wanted to do in terms of the objectives and how I want to meet them, I took into account balance of what I'm interested in, what I like doing. I like doing the research aspects. Doing the coursework I'm not so interested in. But really trying to force myself to come -- go out of my comfort zone of just wanting to do research or just wanting to do mentorship and really, you know, go to lectures and network and do things like that. So that was another thing that I was considering when I was writing my ADP. So I think that's it. So hopefully you guys have lots of questions. Yeates Conwell: Thanks very much. I can never figure out where to hit the applause button. You know, I wanted to. But that was really terrific. Thank have you very much, Amanda. Paula. Paula Clayton: So I just want to tell you something. I think that was excellent and certainly well thought out. When -- I think it was the national alliance. I'm not sure which one of the Washington driven groups but we've been participating in writing kind of a -- the summary of where to go on suicide research. One of them used the CDC term that you use and that is intentional self-injury, and people at AFSP strongly, and I can't tell you how strongly they objected to intentional self-injury to characterize suicide so much so that I hope and I think they've changed terminology in the alliance report. People who've lost someone to suicide, they probably make up 80 percent of our 55 chapters, really don't like to think of the deaths of their loved ones as intentional self-injury. And I think in your career trajectory, you ought to be careful about that in how you title your interests. Amanda Garcia-Williams: Thanks. Yeah, that's good to know I'm not -- I wasn't aware that there was that kind of an objection to that term. Yeates Conwell: That's one of the advantages of the AAS meeting was the opportunity to interact with survivors who are at a stage in their trajectories in which they've thought a lot about this and they are participating in a very interactive and open way that helps us as researchers learn about how they think about and receive the words we use and the way we ask questions and articulate our findings -- Amanda, you want to read the question from Camille there? Jeff has pasted it in there as the last item in the Amanda Garcia-Williams: Sure. Okay, so, Camille asked -- she said, "Great ideas, great trajectory. What is your plan to manage these trajectories? Also, how do you see the PMI fitting into your long term career objectives?" I'm not sure what PMI -- what that stands for. Or maybe I'm blanking. Camille Quinn: I think one of your bullets was about a presidential management internship. Amanda Garcia-Williams: Oh, right, okay. Camille Quinn: Wasn't that one of your bullets? Amanda Garcia-Williams: Yes, yes, I repeat -- yeah, PMF, PMF. Camille Quinn: Oh sorry. Amanda Garcia-Williams: [laughter] That's okay. I was like, yeah -- I was like, oh. I don't really know what's going on. Okay, so in terms of managing my trajectories, primarily most of the work that I'm going to be doing will sort of apply to both of those trajectories, will help me to be able to reach those goals. But ultimately for -- I think for either scenario, you know, having, you know, publications and networking with people and working on a variety of different projects, will be helpful for both cases. And Atlanta is quite -- is not that big, and so networking with people at Georgia State is not completely -- you know, is not so different or difficult than networking with people at CDC. I mean that -- I think it's just -- it's -- I'm doing a lot of the same things to meet both those goals. And in terms of PMF, PMF and EIS I think would -- EIS, maybe more so than PMF, would give me a chance of being already sort of in the sort of -- but not -- I don't know how to phrase the word, but it would give me a leg up in a place like CDC because they are so prestigious, these programs, and the point of the programs are really to create this group of individuals who are ready to become leaders in -- at CDC and you get to choose, sort of some degree, the area that you go into. But it really gives you sort of more notoriety, more sort of stature instead of just coming as like, you know, public health analyst and trying to work your way up. And so I think that that would allow me to become more of a stronger figure, a stronger social scientist at a place like CDC. So I don't know if they answers your question. Yeates Conwell: Unfortunately, we've run a little over and are going to need to stop the day or at least I got to run to another meeting. So if people want to hang around and continue talking for a little while I suspect that's okay. I defer to tell Jeff and Krystyna on that and just to say again thank you very much, Amanda, for bringing us up to date on your thinking. I mean clearly, you just come a long way. I appreciate the bullet points that you provided at the end saying how stressful this was. It certainly is. It's just the investment that we need to make. And you've really obviously hung in there with it. Over time, your decisions will hopefully get a little easier and the way to think about this stuff will continue to evolve. So, again, very, very well done. Our next meeting on this venue is October 3rd, and we're going to have 3 ADP reviews. So we'll be in touch before then. I believe that's it. I got to sign off then. Thank you all. We'll see you again soon.